[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:01:51 dist-test-slave-4h14 krb5kdc[3539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:01:51 dist-test-slave-4h14 krb5kdc[3539](info): set up 2 sockets
Jun 20 06:01:51 dist-test-slave-4h14 krb5kdc[3539](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 06:01:52.606420 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:01:52 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:01:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:01:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:01:52.732432 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43713
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:52.846786 3552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:52.847069 3552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:52.847110 3552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:52.851245 3552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:01:52.851310 3552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:52.851330 3552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:52.851347 3552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:01:52.851362 3552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:01:52.856633 3552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:52.857895 3552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:52.858956 3552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:52.868060 3558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:52.868036 3552 server_base.cc:1034] running on GCE node
W20260620 06:01:52.868036 3557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:52.868024 3560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:52.868963 3552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:52.870153 3552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:52.871346 3552 hybrid_clock.cc:648] HybridClock initialized: now 1781935312871317 us; error 45 us; skew 500 ppm
Jun 20 06:01:52 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935312, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:52.874233 3552 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:01:52.875190 3552 webserver.cc:460] Webserver started at http://127.3.115.62:42835/ using document root <none> and password file <none>
I20260620 06:01:52.875761 3552 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:52.875849 3552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:52.876082 3552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:52.878274 3552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "7e45db4472b74a91a82d5eb8f084d0e4"
format_stamp: "Formatted at 2026-06-20 06:01:52 on dist-test-slave-4h14"
server_key: "7dfedbcdb0803589d1b225468715600f"
server_key_iv: "5ef39237c7931f8f081aff6cfa4b4594"
server_key_version: "encryptionkey@0"
I20260620 06:01:52.878850 3552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "7e45db4472b74a91a82d5eb8f084d0e4"
format_stamp: "Formatted at 2026-06-20 06:01:52 on dist-test-slave-4h14"
server_key: "7dfedbcdb0803589d1b225468715600f"
server_key_iv: "5ef39237c7931f8f081aff6cfa4b4594"
server_key_version: "encryptionkey@0"
I20260620 06:01:52.882630 3552 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:01:52.885214 3567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:52.886471 3552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:01:52.886605 3552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "7e45db4472b74a91a82d5eb8f084d0e4"
format_stamp: "Formatted at 2026-06-20 06:01:52 on dist-test-slave-4h14"
server_key: "7dfedbcdb0803589d1b225468715600f"
server_key_iv: "5ef39237c7931f8f081aff6cfa4b4594"
server_key_version: "encryptionkey@0"
I20260620 06:01:52.886713 3552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:52.904824 3552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:52.908267 3552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:52.908489 3552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:52.915546 3552 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:43713
I20260620 06:01:52.915612 3619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:43713 every 8 connection(s)
I20260620 06:01:52.916544 3552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:01:52.919420 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 3552
I20260620 06:01:52.919572 3620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:52.919585 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:01:52.921252 3532 external_mini_cluster.cc:1428] Setting key 57d4f1e79aaa1fa3fb980f6cad3f4a25
I20260620 06:01:52.926467 3620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4: Bootstrap starting.
I20260620 06:01:52.929250 3620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:52.930157 3620 log.cc:824] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:52.932423 3620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4: No bootstrap required, opened a new log
I20260620 06:01:52.936815 3620 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43713 } }
I20260620 06:01:52.937095 3620 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:52.937177 3620 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e45db4472b74a91a82d5eb8f084d0e4, State: Initialized, Role: FOLLOWER
I20260620 06:01:52.937608 3620 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43713 } }
I20260620 06:01:52.937719 3620 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:01:52.937806 3620 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:01:52.937909 3620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:52.938807 3620 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43713 } }
I20260620 06:01:52.939172 3620 leader_election.cc:302] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e45db4472b74a91a82d5eb8f084d0e4; no voters:
I20260620 06:01:52.939561 3620 leader_election.cc:288] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:01:52.939638 3625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:52.939841 3625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [term 1 LEADER]: Becoming Leader. State: Replica: 7e45db4472b74a91a82d5eb8f084d0e4, State: Running, Role: LEADER
I20260620 06:01:52.940142 3625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43713 } }
I20260620 06:01:52.940881 3620 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 06:01:52 dist-test-slave-4h14 krb5kdc[3539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:52.942907 3626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43713 } } }
I20260620 06:01:52.943054 3626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [sys.catalog]: This master's current role is: LEADER
I20260620 06:01:52.943903 3627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e45db4472b74a91a82d5eb8f084d0e4. Latest consensus state: current_term: 1 leader_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e45db4472b74a91a82d5eb8f084d0e4" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43713 } } }
I20260620 06:01:52.944010 3627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4 [sys.catalog]: This master's current role is: LEADER
W20260620 06:01:52.946167 3639 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:01:52.946254 3639 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:01:52.946348 3641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:01:52.947249 3641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:01:52.949651 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:52.924967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38150 (local address 127.3.115.62:43713)
0620 06:01:52.925459 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:52.925481 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:52.926485 (+ 1004us) server_negotiation.cc:407] Connection header received
0620 06:01:52.930678 (+ 4193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:52.930703 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:52.931127 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:52.931402 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:52.934382 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:52.935308 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:52.937039 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:52.937676 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:52.937908 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:52.942486 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:52.942494 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:52.942514 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:52.942551 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:52.945368 (+ 2817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:52.946067 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:52.946073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:52.946078 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:52.946224 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:52.946671 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:52.946674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:52.946676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:52.947257 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:52.947476 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:52.948331 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 06:01:52.948640 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":148,"threads_started":1}
I20260620 06:01:52.953250 3641 catalog_manager.cc:1347] Generated new cluster ID: 6469dbffe2d34a31918f9d040b4cb0a2
I20260620 06:01:52.953325 3641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:01:52.975982 3641 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:01:52.977039 3641 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:01:52.994160 3641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e45db4472b74a91a82d5eb8f084d0e4: Generated new TSK 0
I20260620 06:01:52.994966 3641 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:01:53.026099 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43713
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:53.137494 3646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:53.137769 3646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:53.137814 3646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:53.142120 3646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:53.142184 3646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:53.142248 3646 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:01:53.147680 3646 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:53.148833 3646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:53.149878 3646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:53.159443 3651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:53.159480 3652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:53.159495 3646 server_base.cc:1034] running on GCE node
W20260620 06:01:53.159422 3654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:53.160069 3646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:53.160776 3646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:53.161934 3646 hybrid_clock.cc:648] HybridClock initialized: now 1781935313161920 us; error 27 us; skew 500 ppm
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:53.171403 3646 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:01:53.172362 3646 webserver.cc:460] Webserver started at http://127.3.115.1:39095/ using document root <none> and password file <none>
I20260620 06:01:53.173017 3646 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:53.173084 3646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:53.173254 3646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:53.175299 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "73a53b5a7abd454ba94f917362a1e644"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "8c1ad7c1dc2b46738c10ef679076638c"
server_key_iv: "249b129d18ccd2daf764158bb00dc463"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.175822 3646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "73a53b5a7abd454ba94f917362a1e644"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "8c1ad7c1dc2b46738c10ef679076638c"
server_key_iv: "249b129d18ccd2daf764158bb00dc463"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.179572 3646 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:01:53.182031 3661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.183269 3646 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:01:53.183391 3646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "73a53b5a7abd454ba94f917362a1e644"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "8c1ad7c1dc2b46738c10ef679076638c"
server_key_iv: "249b129d18ccd2daf764158bb00dc463"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.183526 3646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:53.203689 3646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:53.206985 3646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:53.207194 3646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:53.207815 3646 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:01:53.208750 3646 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:53.208799 3646 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.208887 3646 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:53.208918 3646 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.218273 3646 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:45679
I20260620 06:01:53.218308 3774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:45679 every 8 connection(s)
I20260620 06:01:53.219198 3646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:01:53.222733 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 3646
I20260620 06:01:53.222868 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:01:53.223142 3532 external_mini_cluster.cc:1428] Setting key a630fdebf6016c59a63ac54dba5c49a6
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:01:53.234342 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.221128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:48271 (local address 127.3.115.62:43713)
0620 06:01:53.221265 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:53.221269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:53.222141 (+ 872us) server_negotiation.cc:407] Connection header received
0620 06:01:53.223861 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:53.223864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:53.223911 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:53.224017 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:53.225439 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.225970 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.227153 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.227396 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.227511 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.230334 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:53.230337 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:53.230339 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:53.230365 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:53.231952 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.232473 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.232478 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.232480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.232573 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.233005 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.233008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.233010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.233240 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:53.233348 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:53.233973 (+ 625us) server_negotiation.cc:299] Negotiation successful
0620 06:01:53.234167 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:01:53.235348 3777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.221520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43713 (local address 127.3.115.1:48271)
0620 06:01:53.221970 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:53.222014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:53.223596 (+ 1582us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:53.224213 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:53.224221 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:53.224758 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:53.225281 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:53.225296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.226102 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.226105 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:53.227022 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:53.227031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.227550 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.227555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:53.227730 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.228368 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:53.228385 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:53.230137 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:53.232066 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:53.232073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:53.232089 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:53.232364 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:53.232694 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:53.232698 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:53.232700 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:53.232900 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:53.233358 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:53.233366 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:53.233713 (+ 347us) client_negotiation.cc:769] Sending connection context
0620 06:01:53.233918 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 06:01:53.234271 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":124,"threads_started":1}
I20260620 06:01:53.236642 3775 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43713
I20260620 06:01:53.236959 3775 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:53.237457 3775 heartbeater.cc:511] Master 127.3.115.62:43713 requested a full tablet report, sending...
I20260620 06:01:53.239096 3583 ts_manager.cc:194] Registered new tserver with Master: 73a53b5a7abd454ba94f917362a1e644 (127.3.115.1:45679)
I20260620 06:01:53.240293 3583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:48271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:01:53.252077 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43713
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:53.367959 3780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:53.368201 3780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:53.368244 3780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:53.372196 3780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:53.372251 3780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:53.372310 3780 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:01:53.377496 3780 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:53.378572 3780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:53.379513 3780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:53.389146 3786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:53.389231 3780 server_base.cc:1034] running on GCE node
W20260620 06:01:53.389142 3788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:53.389142 3785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:53.389751 3780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:53.390465 3780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:53.391654 3780 hybrid_clock.cc:648] HybridClock initialized: now 1781935313391640 us; error 26 us; skew 500 ppm
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:53.394629 3780 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:01:53.395539 3780 webserver.cc:460] Webserver started at http://127.3.115.2:46587/ using document root <none> and password file <none>
I20260620 06:01:53.396116 3780 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:53.396209 3780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:53.396464 3780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:53.398738 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "35fb4f30ea28457f869e6961f00668bf"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "5bb15ccf0f0a32d274106a6e62d1c8ab"
server_key_iv: "524e64342d378be5777830020c22ff1a"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.399361 3780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "35fb4f30ea28457f869e6961f00668bf"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "5bb15ccf0f0a32d274106a6e62d1c8ab"
server_key_iv: "524e64342d378be5777830020c22ff1a"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.403303 3780 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:01:53.405900 3795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.407327 3780 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:01:53.407507 3780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "35fb4f30ea28457f869e6961f00668bf"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "5bb15ccf0f0a32d274106a6e62d1c8ab"
server_key_iv: "524e64342d378be5777830020c22ff1a"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.407645 3780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:53.423183 3780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:53.426810 3780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:53.427026 3780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:53.427706 3780 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:01:53.428546 3780 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:53.428618 3780 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.428761 3780 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:53.428815 3780 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.437633 3780 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:38229
I20260620 06:01:53.437655 3908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:38229 every 8 connection(s)
I20260620 06:01:53.438553 3780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:53.448554 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 3780
I20260620 06:01:53.448721 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:01:53.448987 3532 external_mini_cluster.cc:1428] Setting key 719b76e5252018f85e3a404448fbe281
I20260620 06:01:53.453665 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.440328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:59313 (local address 127.3.115.62:43713)
0620 06:01:53.440525 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:53.440530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:53.441537 (+ 1007us) server_negotiation.cc:407] Connection header received
0620 06:01:53.443042 (+ 1505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:53.443045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:53.443091 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:53.443172 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:53.444582 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.445166 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.446335 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.446542 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.446619 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.449368 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:53.449373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:53.449375 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:53.449406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:53.451046 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.451697 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.451699 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.451701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.451775 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.452188 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.452194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.452198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.452499 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:53.452612 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:53.453249 (+ 637us) server_negotiation.cc:299] Negotiation successful
0620 06:01:53.453413 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260620 06:01:53.454424 3911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.440841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43713 (local address 127.3.115.2:59313)
0620 06:01:53.441348 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:53.441401 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:53.442849 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:53.443408 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:53.443417 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:53.443948 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:53.444439 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:53.444453 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.445315 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.445318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:53.446218 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:53.446227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.446667 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.446669 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:53.446809 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.447369 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:53.447385 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:53.449169 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:53.451167 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:53.451176 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:53.451196 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:53.451591 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:53.451870 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:53.451877 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:53.451882 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:53.452082 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:53.452634 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:53.452640 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:53.452990 (+ 350us) client_negotiation.cc:769] Sending connection context
0620 06:01:53.453162 (+ 172us) client_negotiation.cc:240] Negotiation successful
0620 06:01:53.453467 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":183,"threads_started":1}
I20260620 06:01:53.455706 3909 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43713
I20260620 06:01:53.456017 3909 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:53.456573 3909 heartbeater.cc:511] Master 127.3.115.62:43713 requested a full tablet report, sending...
I20260620 06:01:53.457690 3583 ts_manager.cc:194] Registered new tserver with Master: 35fb4f30ea28457f869e6961f00668bf (127.3.115.2:38229)
I20260620 06:01:53.458305 3583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:59313
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:01:53.476310 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43713
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:53.587164 3914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:53.587414 3914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:53.587491 3914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:53.591744 3914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:53.591806 3914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:53.591902 3914 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:01:53.597404 3914 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:53.598501 3914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:53.599565 3914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:53.609700 3922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:53.609723 3919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:53.609769 3920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:53.610015 3914 server_base.cc:1034] running on GCE node
I20260620 06:01:53.610427 3914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:53.611095 3914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:53.612285 3914 hybrid_clock.cc:648] HybridClock initialized: now 1781935313612267 us; error 32 us; skew 500 ppm
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:53.615000 3914 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:01:53.615833 3914 webserver.cc:460] Webserver started at http://127.3.115.3:46833/ using document root <none> and password file <none>
I20260620 06:01:53.616361 3914 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:53.616422 3914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:53.616585 3914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:53.618650 3914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "1f37d38bd4fe44eeb03784a5d0877891"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "4ef3237da208d7b55353cf962cedb8bb"
server_key_iv: "9b624d833a61586009ed7664ee6ce586"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.619141 3914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "1f37d38bd4fe44eeb03784a5d0877891"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "4ef3237da208d7b55353cf962cedb8bb"
server_key_iv: "9b624d833a61586009ed7664ee6ce586"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.622933 3914 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:01:53.625438 3929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.626688 3914 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:01:53.626791 3914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "1f37d38bd4fe44eeb03784a5d0877891"
format_stamp: "Formatted at 2026-06-20 06:01:53 on dist-test-slave-4h14"
server_key: "4ef3237da208d7b55353cf962cedb8bb"
server_key_iv: "9b624d833a61586009ed7664ee6ce586"
server_key_version: "encryptionkey@0"
I20260620 06:01:53.626880 3914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:53.650832 3914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:53.653932 3914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:53.654091 3914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:53.654641 3914 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:01:53.655416 3914 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:53.655462 3914 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.655501 3914 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:53.655516 3914 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:53.664815 3914 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:41721
I20260620 06:01:53.664932 4042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:41721 every 8 connection(s)
I20260620 06:01:53.665799 3914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:01:53.672780 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 3914
I20260620 06:01:53.672885 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:01:53.673166 3532 external_mini_cluster.cc:1428] Setting key 64d909578822fd9f7979e5bc06c79291
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:53.679801 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.667560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:44253 (local address 127.3.115.62:43713)
0620 06:01:53.667717 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:53.667722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:53.668620 (+ 898us) server_negotiation.cc:407] Connection header received
0620 06:01:53.670055 (+ 1435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:53.670057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:53.670106 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:53.670182 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:53.671359 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.671880 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.673198 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.673526 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.673641 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.676728 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:53.676730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:53.676732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:53.676783 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:53.678049 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.678514 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.678516 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.678518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.678589 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.678880 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.678882 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.678883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.679048 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:53.679123 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:53.679579 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 06:01:53.679674 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:01:53.680629 4045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.668002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43713 (local address 127.3.115.3:44253)
0620 06:01:53.668465 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:53.668509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:53.669847 (+ 1338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:53.670362 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:53.670369 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:53.670721 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:53.671228 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:53.671242 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.671987 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.671989 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:53.673079 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:53.673088 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.673736 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.673741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:53.673960 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.674714 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:53.674736 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:53.676534 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:53.678147 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:53.678152 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:53.678165 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:53.678431 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:53.678666 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:53.678668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:53.678670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:53.678802 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:53.679134 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:53.679139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:53.679379 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 06:01:53.679536 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 06:01:53.679785 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":121,"threads_started":1}
I20260620 06:01:53.681864 4043 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43713
I20260620 06:01:53.682088 4043 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:53.682544 4043 heartbeater.cc:511] Master 127.3.115.62:43713 requested a full tablet report, sending...
I20260620 06:01:53.683488 3583 ts_manager.cc:194] Registered new tserver with Master: 1f37d38bd4fe44eeb03784a5d0877891 (127.3.115.3:41721)
I20260620 06:01:53.684020 3583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:44253
I20260620 06:01:53.687716 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:01:53.714077 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.704803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.3.115.1:45679)
0620 06:01:53.705125 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:53.705139 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:53.705163 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:01:53.705277 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:53.705282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:53.705486 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:53.705609 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:53.706273 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.707027 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.707829 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.708211 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.708306 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.710781 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:53.710790 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:53.710805 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:53.710830 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:53.712431 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.712873 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.712878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.712880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.712957 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.713176 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.713178 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.713179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.713414 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:53.713572 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:53.713716 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 06:01:53.713874 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":149,"threads_started":1}
W20260620 06:01:53.715055 3689 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49792
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[3539](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:01:53.740293 4050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:53.732234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.3.115.1:45679)
0620 06:01:53.732381 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:53.732387 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:53.732541 (+ 154us) server_negotiation.cc:407] Connection header received
0620 06:01:53.732698 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:53.732701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:53.732768 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:53.732840 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:53.733499 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.734035 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.734825 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:53.735042 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:53.735116 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:53.737511 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:53.737514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:53.737516 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:53.737538 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:53.738741 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.739110 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.739113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.739114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.739187 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:53.739417 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:53.739420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:53.739421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:53.739581 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:53.739674 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:53.739870 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 06:01:53.739966 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:01:53.741361 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 3646
I20260620 06:01:53.747566 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 3780
I20260620 06:01:53.753088 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 3914
I20260620 06:01:53.758527 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 3552
2026-06-20T06:01:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2216 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[4057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[4057](info): set up 2 sockets
Jun 20 06:01:53 dist-test-slave-4h14 krb5kdc[4057](info): commencing operation
krb5kdc: starting...
W20260620 06:01:54.804999 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:01:54 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:01:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:01:54.925176 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:39703
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:55.033365 4070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:55.033638 4070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:55.033679 4070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:55.037591 4070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:01:55.037644 4070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:55.037663 4070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:55.037680 4070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:01:55.037695 4070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:01:55.042886 4070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:55.043944 4070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:55.045037 4070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:55.053583 4075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.053715 4070 server_base.cc:1034] running on GCE node
W20260620 06:01:55.053586 4078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:55.053596 4076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.054437 4070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:55.055608 4070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:55.056733 4070 hybrid_clock.cc:648] HybridClock initialized: now 1781935315056698 us; error 53 us; skew 500 ppm
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:55.059619 4070 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:01:55.060570 4070 webserver.cc:460] Webserver started at http://127.3.115.62:43191/ using document root <none> and password file <none>
I20260620 06:01:55.061154 4070 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:55.061218 4070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:55.061457 4070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:55.063536 4070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "4afd784a0d89498592064771b6951f23"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "3ab139b05b937eeb6c403650e60d413e"
server_key_iv: "992a198a44c6b47c6d13a07307f7a59c"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.064105 4070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "4afd784a0d89498592064771b6951f23"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "3ab139b05b937eeb6c403650e60d413e"
server_key_iv: "992a198a44c6b47c6d13a07307f7a59c"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.067941 4070 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:01:55.070453 4085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.071683 4070 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:01:55.071820 4070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "4afd784a0d89498592064771b6951f23"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "3ab139b05b937eeb6c403650e60d413e"
server_key_iv: "992a198a44c6b47c6d13a07307f7a59c"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.071933 4070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:55.092458 4070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:55.095733 4070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:55.095990 4070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:55.103423 4070 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:39703
I20260620 06:01:55.103440 4137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:39703 every 8 connection(s)
I20260620 06:01:55.104423 4070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:01:55.107280 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:55.111209 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4070
I20260620 06:01:55.111444 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:01:55.111692 3532 external_mini_cluster.cc:1428] Setting key 109b139a71b954c1466a1c7acc276b14
I20260620 06:01:55.113181 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23: Bootstrap starting.
I20260620 06:01:55.115950 4138 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:55.116904 4138 log.cc:824] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:55.119032 4138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23: No bootstrap required, opened a new log
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:55.123490 4138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afd784a0d89498592064771b6951f23" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39703 } }
I20260620 06:01:55.123800 4138 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:55.123867 4138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4afd784a0d89498592064771b6951f23, State: Initialized, Role: FOLLOWER
I20260620 06:01:55.124356 4138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afd784a0d89498592064771b6951f23" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39703 } }
I20260620 06:01:55.124528 4138 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:01:55.124601 4138 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:01:55.124738 4138 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:55.125635 4138 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afd784a0d89498592064771b6951f23" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39703 } }
I20260620 06:01:55.126041 4138 leader_election.cc:302] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4afd784a0d89498592064771b6951f23; no voters:
I20260620 06:01:55.126430 4138 leader_election.cc:288] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:01:55.126514 4143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:55.126724 4143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [term 1 LEADER]: Becoming Leader. State: Replica: 4afd784a0d89498592064771b6951f23, State: Running, Role: LEADER
I20260620 06:01:55.127080 4143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afd784a0d89498592064771b6951f23" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39703 } }
I20260620 06:01:55.127205 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.113200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42914 (local address 127.3.115.62:39703)
0620 06:01:55.113660 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.113680 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.113723 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:01:55.114599 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.114623 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.115058 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.115424 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:55.116081 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.117530 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.118383 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.119068 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.119248 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.121845 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:55.121855 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:55.121877 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:55.121915 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:55.123933 (+ 2018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.124403 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.124408 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.124414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.124567 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.124915 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.124919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.124921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.125332 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:55.125549 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.125824 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.126115 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":113,"threads_started":1}
I20260620 06:01:55.127755 4138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:01:55.128793 4145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4afd784a0d89498592064771b6951f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afd784a0d89498592064771b6951f23" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39703 } } }
I20260620 06:01:55.128908 4145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [sys.catalog]: This master's current role is: LEADER
I20260620 06:01:55.129184 4144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4afd784a0d89498592064771b6951f23. Latest consensus state: current_term: 1 leader_uuid: "4afd784a0d89498592064771b6951f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4afd784a0d89498592064771b6951f23" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39703 } } }
I20260620 06:01:55.129267 4144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23 [sys.catalog]: This master's current role is: LEADER
I20260620 06:01:55.129851 4152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:01:55.132886 4152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:01:55.138458 4152 catalog_manager.cc:1347] Generated new cluster ID: ea713a6c7f104f8fbb7a6e0ef2145cc2
I20260620 06:01:55.138546 4152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:01:55.159232 4152 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:01:55.160270 4152 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:01:55.164907 4152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4afd784a0d89498592064771b6951f23: Generated new TSK 0
I20260620 06:01:55.165609 4152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:01:55.195818 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39703
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:55.312357 4164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:55.312633 4164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:55.312695 4164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:55.316586 4164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:55.316672 4164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:55.316753 4164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:01:55.322048 4164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:55.323091 4164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:55.324100 4164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:55.333899 4172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.334048 4164 server_base.cc:1034] running on GCE node
W20260620 06:01:55.333900 4169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:55.333925 4170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.334685 4164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:55.335284 4164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:55.336450 4164 hybrid_clock.cc:648] HybridClock initialized: now 1781935315336433 us; error 33 us; skew 500 ppm
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:55.339520 4164 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:01:55.340411 4164 webserver.cc:460] Webserver started at http://127.3.115.1:40787/ using document root <none> and password file <none>
I20260620 06:01:55.341027 4164 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:55.341097 4164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:55.341274 4164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:55.343389 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "ea70b05c109e444dbe5e829763dfa3b1"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "1432587b81d4ac5a617df3409d9a461c"
server_key_iv: "adca93e7cd3c4826523e4b5ffb9c63ed"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.343919 4164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "ea70b05c109e444dbe5e829763dfa3b1"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "1432587b81d4ac5a617df3409d9a461c"
server_key_iv: "adca93e7cd3c4826523e4b5ffb9c63ed"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.348088 4164 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:01:55.350732 4179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.352164 4164 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:01:55.352301 4164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "ea70b05c109e444dbe5e829763dfa3b1"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "1432587b81d4ac5a617df3409d9a461c"
server_key_iv: "adca93e7cd3c4826523e4b5ffb9c63ed"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.352432 4164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:55.367012 4164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:55.370263 4164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:55.370477 4164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:55.371091 4164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:01:55.371973 4164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:55.372032 4164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.372097 4164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:55.372128 4164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.381333 4292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:40577 every 8 connection(s)
I20260620 06:01:55.381341 4164 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:40577
I20260620 06:01:55.382411 4164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:55.392457 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4164
I20260620 06:01:55.392568 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:01:55.392895 3532 external_mini_cluster.cc:1428] Setting key 3e187251abfe86704b57d96ab7b06c36
I20260620 06:01:55.396400 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.384306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:36325 (local address 127.3.115.62:39703)
0620 06:01:55.384455 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.384459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.385327 (+ 868us) server_negotiation.cc:407] Connection header received
0620 06:01:55.386633 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.386636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.386685 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.386761 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:55.387928 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.388470 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.389627 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.389859 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.389940 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.392547 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:55.392553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:55.392556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:55.392589 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:55.394295 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.394813 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.394817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.394819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.394910 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.395260 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.395265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.395267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.395459 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:55.395563 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.396104 (+ 541us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.396238 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:01:55.397673 4295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.384728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:39703 (local address 127.3.115.1:36325)
0620 06:01:55.385168 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:55.385211 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:55.386428 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:55.386927 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:55.386934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:55.387300 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:55.387798 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.387811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.388576 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.388578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.389517 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.389527 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.389989 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.389992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.390133 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.390728 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:55.390744 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:55.392309 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:55.394403 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:55.394409 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:55.394427 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:55.394696 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:55.395004 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:55.395007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:55.395009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:55.395158 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:55.395575 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:55.395583 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:55.395855 (+ 272us) client_negotiation.cc:769] Sending connection context
0620 06:01:55.396054 (+ 199us) client_negotiation.cc:240] Negotiation successful
0620 06:01:55.396409 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":116,"threads_started":1}
I20260620 06:01:55.399154 4293 heartbeater.cc:347] Connected to a master server at 127.3.115.62:39703
I20260620 06:01:55.399449 4293 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:55.400017 4293 heartbeater.cc:511] Master 127.3.115.62:39703 requested a full tablet report, sending...
I20260620 06:01:55.401758 4102 ts_manager.cc:194] Registered new tserver with Master: ea70b05c109e444dbe5e829763dfa3b1 (127.3.115.1:40577)
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:01:55.403000 4102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:36325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:01:55.420611 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39703
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:55.532897 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:55.533170 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:55.533216 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:55.537467 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:55.537568 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:55.537637 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:01:55.543659 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:55.544884 4298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:55.545921 4298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:55.555533 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:55.555557 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:55.555544 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.556191 4298 server_base.cc:1034] running on GCE node
I20260620 06:01:55.556756 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:55.557415 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:55.558617 4298 hybrid_clock.cc:648] HybridClock initialized: now 1781935315558581 us; error 51 us; skew 500 ppm
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:55.561520 4298 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:01:55.562511 4298 webserver.cc:460] Webserver started at http://127.3.115.2:34779/ using document root <none> and password file <none>
I20260620 06:01:55.563099 4298 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:55.563194 4298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:55.563436 4298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:55.565685 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "f45f9cc1d00b4349bd3ea249896a4cbb"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "70c1e355af52899994511e0236dda35c"
server_key_iv: "68fd404ceecb753c2b2680e6467e24ce"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.566269 4298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "f45f9cc1d00b4349bd3ea249896a4cbb"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "70c1e355af52899994511e0236dda35c"
server_key_iv: "68fd404ceecb753c2b2680e6467e24ce"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.570152 4298 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:01:55.572885 4313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.574187 4298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:01:55.574331 4298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "f45f9cc1d00b4349bd3ea249896a4cbb"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "70c1e355af52899994511e0236dda35c"
server_key_iv: "68fd404ceecb753c2b2680e6467e24ce"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.574451 4298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:55.600549 4298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:55.604187 4298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:55.604432 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:55.605159 4298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:01:55.606041 4298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:55.606119 4298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.606209 4298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:55.606259 4298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.615414 4298 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:35979
I20260620 06:01:55.615433 4426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:35979 every 8 connection(s)
I20260620 06:01:55.616390 4298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:01:55.617376 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4298
I20260620 06:01:55.617465 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:01:55.617818 3532 external_mini_cluster.cc:1428] Setting key 5aebc97f8578a3b3be7b34281cf78976
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:01:55.632537 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.618336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:41929 (local address 127.3.115.62:39703)
0620 06:01:55.618507 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.618511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.619505 (+ 994us) server_negotiation.cc:407] Connection header received
0620 06:01:55.621052 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.621055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.621103 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.621177 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:55.622622 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.623148 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.625121 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.625367 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.625464 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.628857 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:55.628863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:55.628867 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:55.628902 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:55.630386 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.630938 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.630941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.630943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.631023 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.631377 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.631381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.631382 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.631583 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:55.631698 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.632232 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.632358 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:01:55.633422 4429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.618851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:39703 (local address 127.3.115.2:41929)
0620 06:01:55.619362 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:55.619413 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:55.620794 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:55.621384 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:55.621396 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:55.621943 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:55.622467 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.622482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.623283 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.623289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.624962 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.624978 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.625546 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.625553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.625806 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.626653 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:55.626676 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:55.628544 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:55.630529 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:55.630534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:55.630551 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:55.630824 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:55.631122 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:55.631125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:55.631127 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:55.631277 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:55.631705 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:55.631716 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:55.631975 (+ 259us) client_negotiation.cc:769] Sending connection context
0620 06:01:55.632167 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 06:01:55.632509 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":109,"threads_started":1}
I20260620 06:01:55.634671 4427 heartbeater.cc:347] Connected to a master server at 127.3.115.62:39703
I20260620 06:01:55.634946 4427 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:55.635452 4427 heartbeater.cc:511] Master 127.3.115.62:39703 requested a full tablet report, sending...
I20260620 06:01:55.636554 4102 ts_manager.cc:194] Registered new tserver with Master: f45f9cc1d00b4349bd3ea249896a4cbb (127.3.115.2:35979)
I20260620 06:01:55.637419 4102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:41929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:01:55.647244 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39703
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:55.760464 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:55.760799 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:55.760848 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:55.764969 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:55.765038 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:55.765100 4432 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:01:55.770459 4432 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39703
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:55.771569 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:55.772619 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:55.782701 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:55.782696 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.782866 4432 server_base.cc:1034] running on GCE node
W20260620 06:01:55.782696 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:55.783651 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:55.784358 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:55.785557 4432 hybrid_clock.cc:648] HybridClock initialized: now 1781935315785515 us; error 56 us; skew 500 ppm
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:55.788333 4432 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:01:55.789336 4432 webserver.cc:460] Webserver started at http://127.3.115.3:33813/ using document root <none> and password file <none>
I20260620 06:01:55.789947 4432 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:55.790040 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:55.790274 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:55.792464 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "c4d038459e334af98e204a089743d777"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "9f4c59f44f6d79d03e6b726b2a8e4b1f"
server_key_iv: "61661350db23010fcce15d3f5a063e68"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.793174 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "c4d038459e334af98e204a089743d777"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "9f4c59f44f6d79d03e6b726b2a8e4b1f"
server_key_iv: "61661350db23010fcce15d3f5a063e68"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.797299 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:01:55.799839 4447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.801119 4432 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:01:55.801251 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "c4d038459e334af98e204a089743d777"
format_stamp: "Formatted at 2026-06-20 06:01:55 on dist-test-slave-4h14"
server_key: "9f4c59f44f6d79d03e6b726b2a8e4b1f"
server_key_iv: "61661350db23010fcce15d3f5a063e68"
server_key_version: "encryptionkey@0"
I20260620 06:01:55.801384 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:55.853137 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:55.856487 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:55.856714 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:55.857386 4432 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:01:55.858220 4432 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:55.858291 4432 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.858363 4432 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:55.858417 4432 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:55.867619 4432 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40365
I20260620 06:01:55.867640 4560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40365 every 8 connection(s)
I20260620 06:01:55.868578 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:01:55.874387 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4432
I20260620 06:01:55.874490 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:01:55.874784 3532 external_mini_cluster.cc:1428] Setting key b56673de654753fa1441584100a46135
Jun 20 06:01:55 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935315, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:55.883600 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.870520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:47645 (local address 127.3.115.62:39703)
0620 06:01:55.870661 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.870664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.871515 (+ 851us) server_negotiation.cc:407] Connection header received
0620 06:01:55.872965 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.872968 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.873030 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.873130 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:55.874342 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.875086 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.876593 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.876950 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.877061 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.879718 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:55.879721 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:55.879723 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:55.879749 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:55.881367 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.881884 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.881888 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.881889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.882001 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.882338 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.882341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.882342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.882545 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:55.882627 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.883271 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.883431 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:01:55.884450 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.870909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:39703 (local address 127.3.115.3:47645)
0620 06:01:55.871354 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:55.871407 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:55.872728 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:55.873309 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:55.873316 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:55.873713 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:55.874195 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.874209 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.875215 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.875219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.876475 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.876485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.877110 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.877114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.877290 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.877911 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:55.877928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:55.879549 (+ 1621us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:55.881479 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:55.881484 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:55.881499 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:55.881769 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:55.882093 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:55.882096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:55.882097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:55.882235 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:55.882725 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:55.882731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:55.883029 (+ 298us) client_negotiation.cc:769] Sending connection context
0620 06:01:55.883206 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 06:01:55.883493 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":114,"threads_started":1}
I20260620 06:01:55.885736 4561 heartbeater.cc:347] Connected to a master server at 127.3.115.62:39703
I20260620 06:01:55.886013 4561 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:55.886757 4561 heartbeater.cc:511] Master 127.3.115.62:39703 requested a full tablet report, sending...
I20260620 06:01:55.888091 4101 ts_manager.cc:194] Registered new tserver with Master: c4d038459e334af98e204a089743d777 (127.3.115.3:40365)
I20260620 06:01:55.888863 4101 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:47645
I20260620 06:01:55.889333 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:01:55.901405 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.893051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42928 (local address 127.3.115.62:39703)
0620 06:01:55.893212 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.893217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.893350 (+ 133us) server_negotiation.cc:407] Connection header received
0620 06:01:55.893430 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.893432 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.893474 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.893544 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:55.894303 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.894829 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.895704 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.895927 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.896018 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.897561 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:55.897564 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:55.897566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:55.897592 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:55.899374 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.899889 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.899892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.899893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.899981 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:55.900330 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:55.900333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:55.900335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:55.900603 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:55.900815 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.901065 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.901187 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 06:01:55.906957 4101 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42928:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:01:55.910128 4101 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:01:55.925128 4576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.919733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40365 (local address 127.0.0.1:48450)
0620 06:01:55.919934 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:55.919947 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:55.920020 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:55.921212 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:55.921215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:55.921233 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:55.921309 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.921315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.922558 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.922563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.923645 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.923656 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.924497 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.924500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.924610 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.924645 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:01:55.924830 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 06:01:55.924961 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":29,"thread_start_us":82,"threads_started":1}
I20260620 06:01:55.925334 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.919338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.3.115.1:40577)
0620 06:01:55.920239 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.920247 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.920272 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:01:55.920346 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.920351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.920551 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.920727 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:55.921096 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.921953 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.923875 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.924618 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.924785 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.924898 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.925060 (+ 162us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.925167 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":55,"threads_started":1}
I20260620 06:01:55.925525 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.919100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59222 (local address 127.3.115.2:35979)
0620 06:01:55.919751 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.919763 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.919788 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:01:55.919982 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.919990 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.920202 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.920355 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:55.920897 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.922107 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.923746 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.924593 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.924738 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.924898 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.925269 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.925369 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":120,"threads_started":1}
I20260620 06:01:55.925176 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.919183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:40577 (local address 127.0.0.1:45812)
0620 06:01:55.919646 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:55.919661 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:55.919757 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:55.920863 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:55.920869 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:55.920886 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:55.920986 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.920990 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.922071 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.922076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.923751 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.923759 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.924810 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.924816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.924919 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.924941 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:01:55.925011 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:01:55.925068 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":241,"thread_start_us":100,"threads_started":1}
I20260620 06:01:55.925760 4577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.920177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.3.115.3:40365)
0620 06:01:55.920769 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:55.920783 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:55.920807 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:01:55.920880 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:55.920885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:55.921069 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:55.921238 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:55.921412 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.922435 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.923789 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.924367 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.924496 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.924656 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:55.925472 (+ 816us) server_negotiation.cc:299] Negotiation successful
0620 06:01:55.925578 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":129,"threads_started":1}
I20260620 06:01:55.925882 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:55.918985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:35979 (local address 127.0.0.1:59222)
0620 06:01:55.919578 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:55.919629 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:55.919748 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:55.920546 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:55.920552 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:55.920574 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:55.920749 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.920757 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.922232 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.922235 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.923592 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:55.923601 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:55.925021 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:55.925024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:55.925114 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:55.925136 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:01:55.925647 (+ 511us) client_negotiation.cc:240] Negotiation successful
0620 06:01:55.925709 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":72,"thread_start_us":108,"threads_started":1}
I20260620 06:01:55.928859 4227 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0dece37a9b4682ad464d53c33d41c1 (DEFAULT_TABLE table=test-table [id=6e230ef3eb944e1d87c00e5b6495128d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:55.928856 4361 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0dece37a9b4682ad464d53c33d41c1 (DEFAULT_TABLE table=test-table [id=6e230ef3eb944e1d87c00e5b6495128d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:55.929898 4361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0dece37a9b4682ad464d53c33d41c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:55.929898 4227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0dece37a9b4682ad464d53c33d41c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:55.934412 4495 tablet_service.cc:1467] Processing CreateTablet for tablet 9d0dece37a9b4682ad464d53c33d41c1 (DEFAULT_TABLE table=test-table [id=6e230ef3eb944e1d87c00e5b6495128d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:55.935437 4495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d0dece37a9b4682ad464d53c33d41c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:55.936774 4578 tablet_bootstrap.cc:492] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb: Bootstrap starting.
I20260620 06:01:55.939141 4579 tablet_bootstrap.cc:492] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1: Bootstrap starting.
I20260620 06:01:55.939641 4578 tablet_bootstrap.cc:654] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:55.940562 4578 log.cc:824] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:55.941151 4579 tablet_bootstrap.cc:654] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:55.942245 4579 log.cc:824] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:55.942727 4581 tablet_bootstrap.cc:492] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: Bootstrap starting.
I20260620 06:01:55.942965 4578 tablet_bootstrap.cc:492] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb: No bootstrap required, opened a new log
I20260620 06:01:55.943166 4578 ts_tablet_manager.cc:1397] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260620 06:01:55.944227 4579 tablet_bootstrap.cc:492] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1: No bootstrap required, opened a new log
I20260620 06:01:55.944423 4579 ts_tablet_manager.cc:1397] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:01:55.944892 4581 tablet_bootstrap.cc:654] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:55.946017 4581 log.cc:824] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:55.947301 4578 raft_consensus.cc:348] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:55.947604 4578 raft_consensus.cc:374] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:55.947662 4578 raft_consensus.cc:729] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f45f9cc1d00b4349bd3ea249896a4cbb, State: Initialized, Role: FOLLOWER
I20260620 06:01:55.947734 4581 tablet_bootstrap.cc:492] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: No bootstrap required, opened a new log
I20260620 06:01:55.947950 4581 ts_tablet_manager.cc:1397] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:01:55.948143 4578 consensus_queue.cc:260] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:55.948894 4579 raft_consensus.cc:348] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:55.949144 4579 raft_consensus.cc:374] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:55.949196 4579 raft_consensus.cc:729] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea70b05c109e444dbe5e829763dfa3b1, State: Initialized, Role: FOLLOWER
I20260620 06:01:55.949235 4578 ts_tablet_manager.cc:1428] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260620 06:01:55.949594 4427 heartbeater.cc:503] Master 127.3.115.62:39703 was elected leader, sending a full tablet report...
I20260620 06:01:55.949697 4579 consensus_queue.cc:260] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:55.950330 4293 heartbeater.cc:503] Master 127.3.115.62:39703 was elected leader, sending a full tablet report...
I20260620 06:01:55.951675 4579 ts_tablet_manager.cc:1428] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260620 06:01:55.951996 4581 raft_consensus.cc:348] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:55.952242 4581 raft_consensus.cc:374] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:55.952311 4581 raft_consensus.cc:729] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4d038459e334af98e204a089743d777, State: Initialized, Role: FOLLOWER
I20260620 06:01:55.952823 4581 consensus_queue.cc:260] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:55.953490 4561 heartbeater.cc:503] Master 127.3.115.62:39703 was elected leader, sending a full tablet report...
I20260620 06:01:55.953859 4581 ts_tablet_manager.cc:1428] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:01:56.022380 4586 raft_consensus.cc:482] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:01:56.022607 4586 raft_consensus.cc:504] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:56.023846 4586 leader_election.cc:288] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f45f9cc1d00b4349bd3ea249896a4cbb (127.3.115.2:35979), ea70b05c109e444dbe5e829763dfa3b1 (127.3.115.1:40577)
I20260620 06:01:56.027500 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.024098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:40577 (local address 127.3.115.3:37685)
0620 06:01:56.024252 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:56.024269 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:56.024365 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:56.024720 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:56.024723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:56.024739 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:56.024850 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:56.024855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.025587 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.025593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:56.026372 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:56.026381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.027046 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.027049 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:56.027156 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.027179 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:01:56.027270 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:01:56.027362 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71,"mutex_wait_us":1}
I20260620 06:01:56.027500 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.024320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:37685 (local address 127.3.115.1:40577)
0620 06:01:56.024466 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:56.024470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:56.024483 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:01:56.024539 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:56.024541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:56.024580 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:56.024667 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:56.024973 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.025478 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.026503 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.026925 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.027045 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.027109 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:56.027286 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 06:01:56.027362 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:01:56.027720 4587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.024139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:35979 (local address 127.3.115.3:52467)
0620 06:01:56.024552 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:56.024563 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:56.024710 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:56.024980 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:56.024982 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:56.024997 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:56.025114 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:56.025119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.025834 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.025838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:56.026607 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:56.026615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.027248 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.027258 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:56.027403 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.027420 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:01:56.027509 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:01:56.027579 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":23,"thread_start_us":132,"threads_started":1}
I20260620 06:01:56.027717 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.024294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:52467 (local address 127.3.115.2:35979)
0620 06:01:56.024441 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:56.024445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:56.024677 (+ 232us) server_negotiation.cc:407] Connection header received
0620 06:01:56.024829 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:56.024832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:56.024884 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:56.024974 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:56.025222 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.025715 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.026733 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.027125 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.027246 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.027303 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:56.027548 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 06:01:56.027616 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:01:56.028388 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0dece37a9b4682ad464d53c33d41c1" candidate_uuid: "c4d038459e334af98e204a089743d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" is_pre_election: true
I20260620 06:01:56.028389 4247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0dece37a9b4682ad464d53c33d41c1" candidate_uuid: "c4d038459e334af98e204a089743d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea70b05c109e444dbe5e829763dfa3b1" is_pre_election: true
I20260620 06:01:56.028748 4381 raft_consensus.cc:2454] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d038459e334af98e204a089743d777 in term 0.
I20260620 06:01:56.028748 4247 raft_consensus.cc:2454] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4d038459e334af98e204a089743d777 in term 0.
I20260620 06:01:56.029273 4451 leader_election.cc:302] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c4d038459e334af98e204a089743d777, ea70b05c109e444dbe5e829763dfa3b1, f45f9cc1d00b4349bd3ea249896a4cbb; no voters:
I20260620 06:01:56.029558 4586 raft_consensus.cc:2781] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:01:56.029659 4586 raft_consensus.cc:482] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:01:56.029726 4586 raft_consensus.cc:3037] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:56.030793 4586 raft_consensus.cc:504] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:56.031158 4586 leader_election.cc:288] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [CANDIDATE]: Term 1 election: Requested vote from peers f45f9cc1d00b4349bd3ea249896a4cbb (127.3.115.2:35979), ea70b05c109e444dbe5e829763dfa3b1 (127.3.115.1:40577)
I20260620 06:01:56.031523 4381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0dece37a9b4682ad464d53c33d41c1" candidate_uuid: "c4d038459e334af98e204a089743d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb"
I20260620 06:01:56.031574 4247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d0dece37a9b4682ad464d53c33d41c1" candidate_uuid: "c4d038459e334af98e204a089743d777" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea70b05c109e444dbe5e829763dfa3b1"
I20260620 06:01:56.031641 4381 raft_consensus.cc:3037] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:56.031735 4247 raft_consensus.cc:3037] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:56.032815 4381 raft_consensus.cc:2454] T 9d0dece37a9b4682ad464d53c33d41c1 P f45f9cc1d00b4349bd3ea249896a4cbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d038459e334af98e204a089743d777 in term 1.
I20260620 06:01:56.032981 4247 raft_consensus.cc:2454] T 9d0dece37a9b4682ad464d53c33d41c1 P ea70b05c109e444dbe5e829763dfa3b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4d038459e334af98e204a089743d777 in term 1.
I20260620 06:01:56.033176 4448 leader_election.cc:302] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4d038459e334af98e204a089743d777, f45f9cc1d00b4349bd3ea249896a4cbb; no voters:
I20260620 06:01:56.033367 4586 raft_consensus.cc:2781] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:56.033617 4586 raft_consensus.cc:686] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [term 1 LEADER]: Becoming Leader. State: Replica: c4d038459e334af98e204a089743d777, State: Running, Role: LEADER
I20260620 06:01:56.033948 4586 consensus_queue.cc:237] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } }
I20260620 06:01:56.037369 4102 catalog_manager.cc:5495] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777 reported cstate change: term changed from 0 to 1, leader changed from <none> to c4d038459e334af98e204a089743d777 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "c4d038459e334af98e204a089743d777" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f45f9cc1d00b4349bd3ea249896a4cbb" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4d038459e334af98e204a089743d777" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40365 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ea70b05c109e444dbe5e829763dfa3b1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 40577 } health_report { overall_health: UNKNOWN } } }
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:01:56.050902 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.041641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.3.115.1:40577)
0620 06:01:56.041804 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:56.041810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:56.041991 (+ 181us) server_negotiation.cc:407] Connection header received
0620 06:01:56.042151 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:56.042154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:56.042201 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:56.042295 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:56.043027 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.043560 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.044434 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.044729 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.044845 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.047482 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:56.047495 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:56.047511 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:56.047536 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:56.049348 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:56.049765 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:56.049770 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:56.049771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:56.049849 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:56.050104 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:56.050106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:56.050107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:56.050345 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:56.050524 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:56.050672 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 06:01:56.050772 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935316, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:01:56.078078 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.069205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.3.115.1:40577)
0620 06:01:56.069395 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:56.069400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:56.069514 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:01:56.069690 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:56.069693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:56.069745 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:56.069834 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:56.070526 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.071057 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.071894 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.072144 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.072271 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.074882 (+ 2611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:56.074886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:56.074888 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:56.074912 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:56.076334 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:56.076867 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:56.076879 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:56.076880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:56.076964 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:56.077279 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:56.077282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:56.077283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:56.077452 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:56.077554 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:56.077833 (+ 279us) server_negotiation.cc:299] Negotiation successful
0620 06:01:56.077941 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:01:56.103677 4575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:56.095339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.3.115.1:40577)
0620 06:01:56.095549 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:56.095552 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:56.095603 (+ 51us) server_negotiation.cc:407] Connection header received
0620 06:01:56.095720 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:56.095723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:56.095766 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:56.095872 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:56.096507 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.097096 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.097901 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:56.098103 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:56.098181 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:56.100726 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:56.100730 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:56.100732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:56.100754 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:56.102071 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:56.102482 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:56.102484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:56.102485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:56.102554 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:56.102844 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:56.102846 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:56.102847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:56.103018 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:56.103146 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:56.103404 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:01:56.103522 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118}
W20260620 06:01:56.105304 4207 tablet_service.cc:2908] Rejecting scan request for tablet 9d0dece37a9b4682ad464d53c33d41c1: Uninitialized: clean time has not yet been initialized
I20260620 06:01:56.107347 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4164
I20260620 06:01:56.113677 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4298
W20260620 06:01:56.119542 4562 tablet.cc:2369] T 9d0dece37a9b4682ad464d53c33d41c1 P c4d038459e334af98e204a089743d777: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:01:56.120786 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4432
I20260620 06:01:56.126183 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4070
2026-06-20T06:01:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2367 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4602](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4602](info): set up 2 sockets
Jun 20 06:01:56 dist-test-slave-4h14 krb5kdc[4602](info): commencing operation
krb5kdc: starting...
W20260620 06:01:57.161062 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:01:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:01:57.283490 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:36633
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:36633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:57.396792 4615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:57.397086 4615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:57.397157 4615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:57.401381 4615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:01:57.401458 4615 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:01:57.401476 4615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:57.401494 4615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:57.401511 4615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:01:57.401569 4615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:01:57.407203 4615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:36633
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:36633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:57.408402 4615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:57.409576 4615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:57.418354 4620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:57.418514 4615 server_base.cc:1034] running on GCE node
W20260620 06:01:57.418358 4621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:57.418795 4623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:57.419392 4615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:57.420562 4615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:57.421774 4615 hybrid_clock.cc:648] HybridClock initialized: now 1781935317421751 us; error 32 us; skew 500 ppm
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:57.424861 4615 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:01:57.426110 4615 webserver.cc:460] Webserver started at http://127.3.115.62:41873/ using document root <none> and password file <none>
I20260620 06:01:57.426694 4615 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:57.426792 4615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:57.427014 4615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:57.429214 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "98ed1d82c2a441d09d45d7b841653b33"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "69309ad56c6c6e2e3f54168993f518b4"
server_key_iv: "659db01283934587685d4ec5e90a4a2d"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.429804 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "98ed1d82c2a441d09d45d7b841653b33"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "69309ad56c6c6e2e3f54168993f518b4"
server_key_iv: "659db01283934587685d4ec5e90a4a2d"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.433579 4615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:01:57.435984 4630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:57.437458 4615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:01:57.437618 4615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "98ed1d82c2a441d09d45d7b841653b33"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "69309ad56c6c6e2e3f54168993f518b4"
server_key_iv: "659db01283934587685d4ec5e90a4a2d"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.437745 4615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:57.463793 4615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:57.467708 4615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:57.467947 4615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:57.476704 4615 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:36633
I20260620 06:01:57.476727 4692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:36633 every 8 connection(s)
I20260620 06:01:57.477804 4615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:01:57.481343 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4615
I20260620 06:01:57.481405 4693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:57.481487 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:01:57.481750 3532 external_mini_cluster.cc:1428] Setting key 431ab0ff46464404157e3ca3b9df329e
I20260620 06:01:57.487867 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: Bootstrap starting.
I20260620 06:01:57.490204 4693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:57.490981 4693 log.cc:824] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:57.493373 4693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: No bootstrap required, opened a new log
I20260620 06:01:57.497071 4693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ed1d82c2a441d09d45d7b841653b33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36633 } }
I20260620 06:01:57.497329 4693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:57.497426 4693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98ed1d82c2a441d09d45d7b841653b33, State: Initialized, Role: FOLLOWER
I20260620 06:01:57.497541 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.483155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.3.115.62:36633)
0620 06:01:57.483641 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:57.483671 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:57.483803 (+ 132us) server_negotiation.cc:407] Connection header received
0620 06:01:57.484649 (+ 846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:57.484720 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:57.485125 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:57.485498 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:57.486366 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.487436 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.488444 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.489036 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.489222 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.492161 (+ 2939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:57.492167 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:57.492181 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:57.492207 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:57.494444 (+ 2237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.494965 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.494970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.494975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.495090 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.495428 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.495431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.495433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.495828 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:57.495986 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:57.496296 (+ 310us) server_negotiation.cc:299] Negotiation successful
0620 06:01:57.496602 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":122,"threads_started":1}
I20260620 06:01:57.497984 4693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ed1d82c2a441d09d45d7b841653b33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36633 } }
I20260620 06:01:57.498140 4693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:01:57.498229 4693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:01:57.498329 4693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:57.499468 4693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ed1d82c2a441d09d45d7b841653b33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36633 } }
I20260620 06:01:57.499819 4693 leader_election.cc:302] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98ed1d82c2a441d09d45d7b841653b33; no voters:
I20260620 06:01:57.500169 4693 leader_election.cc:288] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:01:57.500250 4698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:57.500512 4698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [term 1 LEADER]: Becoming Leader. State: Replica: 98ed1d82c2a441d09d45d7b841653b33, State: Running, Role: LEADER
I20260620 06:01:57.500880 4698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ed1d82c2a441d09d45d7b841653b33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36633 } }
I20260620 06:01:57.501420 4693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:01:57.502676 4700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98ed1d82c2a441d09d45d7b841653b33. Latest consensus state: current_term: 1 leader_uuid: "98ed1d82c2a441d09d45d7b841653b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ed1d82c2a441d09d45d7b841653b33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36633 } } }
I20260620 06:01:57.502704 4699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98ed1d82c2a441d09d45d7b841653b33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98ed1d82c2a441d09d45d7b841653b33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36633 } } }
I20260620 06:01:57.502801 4700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [sys.catalog]: This master's current role is: LEADER
I20260620 06:01:57.502802 4699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33 [sys.catalog]: This master's current role is: LEADER
I20260620 06:01:57.503791 4707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:01:57.507169 4707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:01:57.513091 4707 catalog_manager.cc:1347] Generated new cluster ID: 7cea299f2dba40a1b969a60ca01ced18
I20260620 06:01:57.513177 4707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:01:57.529022 4707 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:01:57.530339 4707 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:01:57.536303 4707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: Generated new TSK 0
I20260620 06:01:57.537213 4707 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:01:57.567001 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36633
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:57.682284 4719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:57.682597 4719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:57.682678 4719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:57.687479 4719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:57.687582 4719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:57.687654 4719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:01:57.687726 4719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:01:57.693233 4719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:57.694394 4719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:57.695436 4719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:57.705248 4724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:57.705266 4727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:57.705266 4725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:57.705760 4719 server_base.cc:1034] running on GCE node
I20260620 06:01:57.706169 4719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:57.706825 4719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:57.708011 4719 hybrid_clock.cc:648] HybridClock initialized: now 1781935317707998 us; error 26 us; skew 500 ppm
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:57.711134 4719 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:01:57.712247 4719 webserver.cc:460] Webserver started at http://127.3.115.1:46075/ using document root <none> and password file <none>
I20260620 06:01:57.713006 4719 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:57.713095 4719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:57.713335 4719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:57.715610 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "f08355d8734a45d3918c8bf6c6f68892"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "31da2c2a236c5b9a9fc22f95f608375f"
server_key_iv: "57f5d8044dc560c0438de9055dcf3d5b"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.716138 4719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "f08355d8734a45d3918c8bf6c6f68892"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "31da2c2a236c5b9a9fc22f95f608375f"
server_key_iv: "57f5d8044dc560c0438de9055dcf3d5b"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.719942 4719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:01:57.722409 4734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:57.723723 4719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:01:57.723831 4719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "f08355d8734a45d3918c8bf6c6f68892"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "31da2c2a236c5b9a9fc22f95f608375f"
server_key_iv: "57f5d8044dc560c0438de9055dcf3d5b"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.723918 4719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:57.735297 4719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:57.738211 4719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:57.738375 4719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:57.739751 4719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:57.739801 4719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:57.739843 4719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:57.739910 4719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:57.751382 4719 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35843
I20260620 06:01:57.752473 4849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35843 every 8 connection(s)
I20260620 06:01:57.752607 4719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:57.753626 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4719
I20260620 06:01:57.753854 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:01:57.754192 3532 external_mini_cluster.cc:1428] Setting key 1bf00600094671b0b5e805bfdc221d75
I20260620 06:01:57.758313 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.740912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:52395 (local address 127.3.115.62:36633)
0620 06:01:57.741044 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:57.741053 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:57.742250 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 06:01:57.743627 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:57.743632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:57.743701 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:57.743831 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:57.745322 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.746270 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.748455 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.748885 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.749022 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.753820 (+ 4798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:57.753827 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:57.753832 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:57.753867 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:57.755807 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.756480 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.756487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.756491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.756585 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.757048 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.757054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.757059 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.757398 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:57.757504 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:57.758025 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 06:01:57.758154 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:01:57.759317 4743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.741475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.3.115.1:52395)
0620 06:01:57.742075 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:57.742130 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:57.743241 (+ 1111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:57.744038 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:57.744045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:57.744550 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:57.745172 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.745186 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.746412 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.746416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.748321 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.748336 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.750352 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.750357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.750621 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.751390 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:57.751412 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:57.753585 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:57.755911 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.755916 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.755931 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.756334 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.756730 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.756734 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.756735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.756926 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.757516 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:57.757522 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:57.757764 (+ 242us) client_negotiation.cc:769] Sending connection context
0620 06:01:57.757958 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 06:01:57.758314 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":143,"threads_started":1}
I20260620 06:01:57.760910 4850 heartbeater.cc:347] Connected to a master server at 127.3.115.62:36633
I20260620 06:01:57.761083 4850 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:57.761734 4850 heartbeater.cc:511] Master 127.3.115.62:36633 requested a full tablet report, sending...
I20260620 06:01:57.763506 4646 ts_manager.cc:194] Registered new tserver with Master: f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843)
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:01:57.764931 4646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:52395
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:01:57.771170 4857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.763093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.3.115.1:47899)
0620 06:01:57.763432 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:57.763446 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:57.763549 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:57.763859 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:57.763862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:57.764073 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:57.764191 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.764198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.765303 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.765306 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.766110 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.766121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.766702 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.766706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.766792 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.767435 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:57.767449 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:57.767815 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:57.769591 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.769597 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.769601 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.769877 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.770181 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.770184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.770186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.770289 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.770728 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:57.770731 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:57.770809 (+ 78us) client_negotiation.cc:769] Sending connection context
0620 06:01:57.770901 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 06:01:57.771015 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":118,"threads_started":1}
I20260620 06:01:57.771389 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.763165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:47899 (local address 127.3.115.62:36633)
0620 06:01:57.763281 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:57.763285 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:57.763536 (+ 251us) server_negotiation.cc:407] Connection header received
0620 06:01:57.763644 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:57.763648 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:57.763716 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:57.763820 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:57.764330 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.765172 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.766264 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.766572 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.766666 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.767954 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:57.767957 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:57.767960 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:57.767981 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:57.769484 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.769999 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.770002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.770003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.770081 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.770396 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.770398 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.770399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.770613 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:57.770720 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:57.770987 (+ 267us) server_negotiation.cc:299] Negotiation successful
0620 06:01:57.771130 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:01:57.785223 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36633
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:57.896522 4859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:57.896871 4859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:57.896950 4859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:57.901077 4859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:57.901180 4859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:57.901257 4859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:01:57.901350 4859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:01:57.907054 4859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:57.908205 4859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:57.909319 4859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:01:57.919222 4859 server_base.cc:1034] running on GCE node
W20260620 06:01:57.919268 4867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:57.919400 4864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:57.919268 4865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:57.919952 4859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:57.920733 4859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:57.921948 4859 hybrid_clock.cc:648] HybridClock initialized: now 1781935317921934 us; error 32 us; skew 500 ppm
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:57.924993 4859 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:01:57.925977 4859 webserver.cc:460] Webserver started at http://127.3.115.2:35623/ using document root <none> and password file <none>
I20260620 06:01:57.926587 4859 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:57.926682 4859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:57.926926 4859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:57.929248 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "228187c39a294d7b86852abf364e871c"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "a41d592124d8cf2f94f96b26d426145a"
server_key_iv: "0ef05b09012b1fd820b546d1ccbff5af"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.929838 4859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "228187c39a294d7b86852abf364e871c"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "a41d592124d8cf2f94f96b26d426145a"
server_key_iv: "0ef05b09012b1fd820b546d1ccbff5af"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.933660 4859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:01:57.936260 4874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:57.937647 4859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:01:57.937836 4859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "228187c39a294d7b86852abf364e871c"
format_stamp: "Formatted at 2026-06-20 06:01:57 on dist-test-slave-4h14"
server_key: "a41d592124d8cf2f94f96b26d426145a"
server_key_iv: "0ef05b09012b1fd820b546d1ccbff5af"
server_key_version: "encryptionkey@0"
I20260620 06:01:57.937952 4859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:57.958971 4859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:57.962541 4859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:57.962786 4859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:57.964376 4859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:57.964427 4859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:57.964468 4859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:57.964509 4859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:01:57 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:57.975131 4859 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:34977
I20260620 06:01:57.975173 4989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:34977 every 8 connection(s)
I20260620 06:01:57.976356 4859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:01:57.979151 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.965284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:45903 (local address 127.3.115.62:36633)
0620 06:01:57.965473 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:57.965479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:57.966297 (+ 818us) server_negotiation.cc:407] Connection header received
0620 06:01:57.967488 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:57.967491 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:57.967555 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:57.967667 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:57.969403 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.970298 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.972200 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.972412 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.972505 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.975526 (+ 3021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:57.975530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:57.975533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:57.975597 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:57.977308 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.977792 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.977794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.977795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.977871 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.978150 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.978152 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.978153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.978360 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:57.978435 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:57.978923 (+ 488us) server_negotiation.cc:299] Negotiation successful
0620 06:01:57.979030 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:01:57.979946 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.965685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.3.115.2:45903)
0620 06:01:57.966133 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:57.966182 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:57.967218 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:57.967903 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:57.967916 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:57.968425 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:57.969218 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.969243 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.970452 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.970458 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.972077 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.972086 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.972600 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.972607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.972869 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.973616 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:57.973640 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:57.975312 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:57.977441 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.977445 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.977457 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.977707 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.977949 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.977951 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.977952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.978076 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.978460 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:57.978465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:57.978686 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:01:57.978839 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:01:57.979086 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":121,"threads_started":1}
I20260620 06:01:57.981211 4990 heartbeater.cc:347] Connected to a master server at 127.3.115.62:36633
I20260620 06:01:57.981365 4990 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:57.981885 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4859
I20260620 06:01:57.981935 4990 heartbeater.cc:511] Master 127.3.115.62:36633 requested a full tablet report, sending...
I20260620 06:01:57.982009 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:01:57.982306 3532 external_mini_cluster.cc:1428] Setting key 8e37730b0ef2e505bed3410cfe0c3e70
I20260620 06:01:57.982995 4647 ts_manager.cc:194] Registered new tserver with Master: 228187c39a294d7b86852abf364e871c (127.3.115.2:34977)
I20260620 06:01:57.983745 4647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:45903
I20260620 06:01:57.992048 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.983528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:36003 (local address 127.3.115.62:36633)
0620 06:01:57.983634 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:57.983638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:57.983875 (+ 237us) server_negotiation.cc:407] Connection header received
0620 06:01:57.984009 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:57.984014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:57.984065 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:57.984133 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:57.984749 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.985559 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.986844 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.987182 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.987294 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.988708 (+ 1414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:57.988712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:57.988715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:57.988745 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:57.990232 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.990770 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.990773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.990774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.990849 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:57.991140 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:57.991142 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:57.991143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:57.991376 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:57.991460 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:57.991779 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 06:01:57.991873 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:01:57.992108 4996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:57.983477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.3.115.2:36003)
0620 06:01:57.983802 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:57.983819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:57.983914 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:57.984238 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:57.984241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:57.984463 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:57.984578 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.984585 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.985705 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.985709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.986700 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:57.986714 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:57.987302 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:57.987308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:57.987447 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:57.987987 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:57.988007 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:57.988491 (+ 484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:57.990359 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.990366 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.990371 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.990664 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.990938 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:57.990943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:57.990948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:57.991051 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:57.991483 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:57.991488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:57.991605 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 06:01:57.991728 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 06:01:57.991889 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:01:58.012269 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36633
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:01:58.124785 4999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:01:58.125084 4999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:01:58.125159 4999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:01:58.130211 4999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:01:58.130348 4999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:01:58.130415 4999 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:01:58.130483 4999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:01:58.139070 4999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:36633
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:01:58.140588 4999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:01:58.142130 4999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:01:58.155365 5005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:01:58.155364 5004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:58.155427 4999 server_base.cc:1034] running on GCE node
W20260620 06:01:58.155763 5007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:01:58.156279 4999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:01:58.157266 4999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:01:58.158465 4999 hybrid_clock.cc:648] HybridClock initialized: now 1781935318158437 us; error 50 us; skew 500 ppm
Jun 20 06:01:58 dist-test-slave-4h14 krb5kdc[4602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:01:58.162473 4999 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:01:58.163859 4999 webserver.cc:460] Webserver started at http://127.3.115.3:33695/ using document root <none> and password file <none>
I20260620 06:01:58.164714 4999 fs_manager.cc:359] Metadata directory not provided
I20260620 06:01:58.164836 4999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:01:58.165127 4999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:01:58.168202 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "6b9edad54daa469caa664940947f1e46"
format_stamp: "Formatted at 2026-06-20 06:01:58 on dist-test-slave-4h14"
server_key: "fff999e5295d3ea9c5ec209d40c2d734"
server_key_iv: "13cdae1032c60dbf6d3ddf91c6862b16"
server_key_version: "encryptionkey@0"
I20260620 06:01:58.169029 4999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "6b9edad54daa469caa664940947f1e46"
format_stamp: "Formatted at 2026-06-20 06:01:58 on dist-test-slave-4h14"
server_key: "fff999e5295d3ea9c5ec209d40c2d734"
server_key_iv: "13cdae1032c60dbf6d3ddf91c6862b16"
server_key_version: "encryptionkey@0"
I20260620 06:01:58.174563 4999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:01:58.177882 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:58.179558 4999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:01:58.179708 4999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "6b9edad54daa469caa664940947f1e46"
format_stamp: "Formatted at 2026-06-20 06:01:58 on dist-test-slave-4h14"
server_key: "fff999e5295d3ea9c5ec209d40c2d734"
server_key_iv: "13cdae1032c60dbf6d3ddf91c6862b16"
server_key_version: "encryptionkey@0"
I20260620 06:01:58.179853 4999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:01:58.216403 4999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:01:58.219647 4999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:01:58.219807 4999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:01:58.221470 4999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:01:58.221522 4999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:01:58.221563 4999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:01:58.221617 4999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:01:58 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935318, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:58.233055 4999 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:42783
I20260620 06:01:58.233127 5129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:42783 every 8 connection(s)
I20260620 06:01:58.233985 4999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:01:58.235498 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.222335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:51021 (local address 127.3.115.62:36633)
0620 06:01:58.222471 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.222475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.223524 (+ 1049us) server_negotiation.cc:407] Connection header received
0620 06:01:58.224645 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.224652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.224745 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.224842 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:58.226304 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.227203 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.228549 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.228916 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.229023 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.231725 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:58.231728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:58.231731 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:58.231760 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:58.233450 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.233959 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.233964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.233966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.234056 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.234472 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.234475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.234477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.234704 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:58.234809 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.235276 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.235382 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:01:58.236337 5023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.222851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.3.115.3:51021)
0620 06:01:58.223363 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.223405 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.224406 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.225045 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.225053 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.225557 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:58.226136 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.226160 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.227339 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.227346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.228423 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.228440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.229070 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.229074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.229280 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.229988 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:58.230005 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:58.231552 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:58.233569 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:58.233575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:58.233591 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:58.233853 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:58.234171 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:58.234175 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:58.234177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:58.234365 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:58.234821 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:58.234827 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:58.235061 (+ 234us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.235217 (+ 156us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.235462 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":132,"threads_started":1}
I20260620 06:01:58.237659 5130 heartbeater.cc:347] Connected to a master server at 127.3.115.62:36633
I20260620 06:01:58.237813 5130 heartbeater.cc:464] Registering TS with master...
I20260620 06:01:58.238332 5130 heartbeater.cc:511] Master 127.3.115.62:36633 requested a full tablet report, sending...
I20260620 06:01:58.239486 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 4999
I20260620 06:01:58.239506 4646 ts_manager.cc:194] Registered new tserver with Master: 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783)
I20260620 06:01:58.239609 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:01:58.239869 3532 external_mini_cluster.cc:1428] Setting key d5d3b3cf03771483efc60ab76ae8fd1e
I20260620 06:01:58.240161 4646 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:51021
I20260620 06:01:58.242301 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:01:58.250113 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.240495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:43971 (local address 127.3.115.62:36633)
0620 06:01:58.240611 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.240617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.240920 (+ 303us) server_negotiation.cc:407] Connection header received
0620 06:01:58.241294 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.241297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.241360 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.241445 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:58.242546 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.243324 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.244362 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.244594 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.244747 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.246254 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:58.246260 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:58.246267 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:58.246299 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:58.248196 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.248811 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.248814 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.248815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.248897 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.249217 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.249220 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.249221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.249404 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:58.249528 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.249847 (+ 319us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.249955 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:01:58.250119 5136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.240408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.3.115.3:43971)
0620 06:01:58.240829 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.240863 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.241179 (+ 316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.242085 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.242088 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.242312 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:58.242427 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.242435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.243492 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.243494 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.244225 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.244240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.244760 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.244766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.244958 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.245504 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:58.245524 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:58.246089 (+ 565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:58.248330 (+ 2241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:58.248333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:58.248359 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:58.248649 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:58.249001 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:58.249007 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:58.249010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:58.249117 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:58.249521 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:58.249524 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:58.249627 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.249738 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.249901 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":102,"threads_started":1}
Jun 20 06:01:58 dist-test-slave-4h14 krb5kdc[4602](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:01:58 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:58.270597 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.262177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46436 (local address 127.3.115.62:36633)
0620 06:01:58.262310 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.262314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.262326 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:01:58.262390 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.262392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.262431 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.262499 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:58.263254 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.263802 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.264620 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.264891 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.264970 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.267372 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:58.267375 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:58.267377 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:58.267403 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:58.268930 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.269356 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.269362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.269363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.269442 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.269721 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.269726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.269728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.269916 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:58.270053 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.270309 (+ 256us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.270432 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:01:58.273566 4647 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:01:58.276585 4647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:01:58.291203 5147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.284972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35843 (local address 127.0.0.1:55000)
0620 06:01:58.285711 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.285764 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.285917 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.286714 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.286720 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.286742 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.286898 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.286905 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.288323 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.288326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.289786 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.289798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.290622 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.290626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.290739 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.290780 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.290897 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.291027 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":117,"thread_start_us":77,"threads_started":1}
I20260620 06:01:58.291191 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.285069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.3.115.1:35843)
0620 06:01:58.285493 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.285501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.286076 (+ 575us) server_negotiation.cc:407] Connection header received
0620 06:01:58.286159 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.286166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.286391 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.286543 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.287039 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.288218 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.289951 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.290488 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.290626 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.290789 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.290940 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.291040 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":257,"threads_started":1}
I20260620 06:01:58.291247 5148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.285015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:42783 (local address 127.0.0.1:38140)
0620 06:01:58.285963 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.285978 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.286062 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.286998 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.287001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.287015 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.287191 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.287198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.288247 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.288253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.289787 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.289798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.290822 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.290826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.290914 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.290937 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.290993 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.291066 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":880,"mutex_wait_us":100,"thread_start_us":89,"threads_started":1}
I20260620 06:01:58.291708 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.285905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.3.115.2:34977)
0620 06:01:58.286304 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.286320 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.286640 (+ 320us) server_negotiation.cc:407] Connection header received
0620 06:01:58.286719 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.286724 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.286939 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.287092 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.287284 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.288371 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.289811 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.290671 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.290795 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.290980 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.291471 (+ 491us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.291575 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":113,"threads_started":1}
I20260620 06:01:58.291867 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.286236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38140 (local address 127.3.115.3:42783)
0620 06:01:58.286504 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.286515 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.286537 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:01:58.286609 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.286614 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.286826 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.287339 (+ 513us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.287448 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.288135 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.289913 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.290671 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.291366 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.291533 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.291578 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.291661 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":99,"threads_started":1}
I20260620 06:01:58.291384 5150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.285732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:34977 (local address 127.0.0.1:37702)
0620 06:01:58.286458 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.286472 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.286553 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.287072 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.287075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.287087 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.287194 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.287198 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.288477 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.288482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.289675 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.289684 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.291036 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.291040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.291116 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.291137 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.291200 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.291258 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"mutex_wait_us":45,"thread_start_us":94,"threads_started":1}
I20260620 06:01:58.294698 5064 tablet_service.cc:1467] Processing CreateTablet for tablet c405ea29160a434889661589c93123ce (DEFAULT_TABLE table=test-table [id=3e08fb7f203348b88932a3dd3a623423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:58.294714 4924 tablet_service.cc:1467] Processing CreateTablet for tablet c405ea29160a434889661589c93123ce (DEFAULT_TABLE table=test-table [id=3e08fb7f203348b88932a3dd3a623423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:58.294703 4784 tablet_service.cc:1467] Processing CreateTablet for tablet c405ea29160a434889661589c93123ce (DEFAULT_TABLE table=test-table [id=3e08fb7f203348b88932a3dd3a623423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:58.295631 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c405ea29160a434889661589c93123ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.295680 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c405ea29160a434889661589c93123ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.295699 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c405ea29160a434889661589c93123ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.302886 5153 tablet_bootstrap.cc:492] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Bootstrap starting.
I20260620 06:01:58.304199 5154 tablet_bootstrap.cc:492] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Bootstrap starting.
I20260620 06:01:58.305025 5155 tablet_bootstrap.cc:492] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Bootstrap starting.
I20260620 06:01:58.305382 5153 tablet_bootstrap.cc:654] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.306120 5153 log.cc:824] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:58.306294 5154 tablet_bootstrap.cc:654] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.307034 5155 tablet_bootstrap.cc:654] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.307106 5154 log.cc:824] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:58.307790 5155 log.cc:824] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:01:58.308423 5153 tablet_bootstrap.cc:492] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: No bootstrap required, opened a new log
I20260620 06:01:58.308645 5153 ts_tablet_manager.cc:1397] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:01:58.309131 5154 tablet_bootstrap.cc:492] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: No bootstrap required, opened a new log
I20260620 06:01:58.309329 5154 ts_tablet_manager.cc:1397] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:01:58.309702 5155 tablet_bootstrap.cc:492] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: No bootstrap required, opened a new log
I20260620 06:01:58.309911 5155 ts_tablet_manager.cc:1397] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:01:58.312237 5154 raft_consensus.cc:348] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.312487 5154 raft_consensus.cc:374] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.312548 5154 raft_consensus.cc:729] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9edad54daa469caa664940947f1e46, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.313073 5154 consensus_queue.cc:260] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.313304 5153 raft_consensus.cc:348] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.313572 5153 raft_consensus.cc:374] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.313629 5153 raft_consensus.cc:729] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08355d8734a45d3918c8bf6c6f68892, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.313750 5155 raft_consensus.cc:348] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.313987 5155 raft_consensus.cc:374] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.314043 5155 raft_consensus.cc:729] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228187c39a294d7b86852abf364e871c, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.314033 5153 consensus_queue.cc:260] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.314435 5155 consensus_queue.cc:260] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.314935 5153 ts_tablet_manager.cc:1428] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 06:01:58.314935 5154 ts_tablet_manager.cc:1428] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:01:58.315164 4990 heartbeater.cc:503] Master 127.3.115.62:36633 was elected leader, sending a full tablet report...
I20260620 06:01:58.315219 5130 heartbeater.cc:503] Master 127.3.115.62:36633 was elected leader, sending a full tablet report...
I20260620 06:01:58.315267 4850 heartbeater.cc:503] Master 127.3.115.62:36633 was elected leader, sending a full tablet report...
I20260620 06:01:58.315291 5155 ts_tablet_manager.cc:1428] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:01:58.342955 5161 raft_consensus.cc:482] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:01:58.343187 5161 raft_consensus.cc:504] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.344419 5161 leader_election.cc:288] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843), 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783)
I20260620 06:01:58.349061 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.344878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:58755 (local address 127.3.115.1:35843)
0620 06:01:58.345016 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.345020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.345041 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:01:58.345117 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.345120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.345174 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.345264 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.345619 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.346485 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.347841 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.348489 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.348618 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.348747 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.348894 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.348958 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:01:58.349087 4883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.344757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35843 (local address 127.3.115.2:58755)
0620 06:01:58.344858 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.344873 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.344976 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.345330 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.345333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.345353 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.345474 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.345482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.346596 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.346600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.347721 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.347729 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.348634 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.348640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.348767 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.348793 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.348874 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.348947 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":23}
I20260620 06:01:58.349810 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405ea29160a434889661589c93123ce" candidate_uuid: "228187c39a294d7b86852abf364e871c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08355d8734a45d3918c8bf6c6f68892" is_pre_election: true
I20260620 06:01:58.349956 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.344757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:42783 (local address 127.3.115.2:57237)
0620 06:01:58.345274 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.345287 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.345442 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.345712 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.345715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.345732 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.346336 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.346346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.347364 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.347367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.348550 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.348562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.349525 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.349528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.349647 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.349668 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.349741 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.349803 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":148,"thread_start_us":145,"threads_started":1}
I20260620 06:01:58.350042 5152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.344838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:57237 (local address 127.3.115.3:42783)
0620 06:01:58.344972 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.344977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.345374 (+ 397us) server_negotiation.cc:407] Connection header received
0620 06:01:58.345559 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.345562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.345614 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.345698 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.346466 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.347235 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.348700 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.349280 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.349379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.349438 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.349860 (+ 422us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.349937 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:01:58.350174 4804 raft_consensus.cc:2454] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228187c39a294d7b86852abf364e871c in term 0.
I20260620 06:01:58.350718 4877 leader_election.cc:302] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228187c39a294d7b86852abf364e871c, f08355d8734a45d3918c8bf6c6f68892; no voters:
I20260620 06:01:58.350755 5084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405ea29160a434889661589c93123ce" candidate_uuid: "228187c39a294d7b86852abf364e871c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9edad54daa469caa664940947f1e46" is_pre_election: true
I20260620 06:01:58.351002 5161 raft_consensus.cc:2781] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:01:58.351094 5161 raft_consensus.cc:482] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:01:58.351073 5084 raft_consensus.cc:2454] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 228187c39a294d7b86852abf364e871c in term 0.
I20260620 06:01:58.351150 5161 raft_consensus.cc:3037] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.352329 5161 raft_consensus.cc:504] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.352754 5161 leader_election.cc:288] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [CANDIDATE]: Term 1 election: Requested vote from peers f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843), 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783)
I20260620 06:01:58.353104 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405ea29160a434889661589c93123ce" candidate_uuid: "228187c39a294d7b86852abf364e871c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08355d8734a45d3918c8bf6c6f68892"
I20260620 06:01:58.353127 5084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c405ea29160a434889661589c93123ce" candidate_uuid: "228187c39a294d7b86852abf364e871c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9edad54daa469caa664940947f1e46"
I20260620 06:01:58.353253 4804 raft_consensus.cc:3037] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.353261 5084 raft_consensus.cc:3037] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.354305 4804 raft_consensus.cc:2454] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228187c39a294d7b86852abf364e871c in term 1.
I20260620 06:01:58.354650 4877 leader_election.cc:302] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 228187c39a294d7b86852abf364e871c, f08355d8734a45d3918c8bf6c6f68892; no voters:
I20260620 06:01:58.354671 5084 raft_consensus.cc:2454] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 228187c39a294d7b86852abf364e871c in term 1.
I20260620 06:01:58.354847 5161 raft_consensus.cc:2781] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:58.355082 5161 raft_consensus.cc:686] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 1 LEADER]: Becoming Leader. State: Replica: 228187c39a294d7b86852abf364e871c, State: Running, Role: LEADER
I20260620 06:01:58.355425 5161 consensus_queue.cc:237] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } }
I20260620 06:01:58.358850 4646 catalog_manager.cc:5495] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c reported cstate change: term changed from 0 to 1, leader changed from <none> to 228187c39a294d7b86852abf364e871c (127.3.115.2). New cstate: current_term: 1 leader_uuid: "228187c39a294d7b86852abf364e871c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } health_report { overall_health: HEALTHY } } }
I20260620 06:01:58.374938 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.371100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37708 (local address 127.3.115.2:34977)
0620 06:01:58.371242 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.371247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.371261 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:01:58.371460 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.371463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.371528 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.371628 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:01:58.371916 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.372432 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.373402 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.373648 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.373755 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.374133 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.374535 (+ 402us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.374660 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.374741 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.374795 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:01:58.380424 5084 raft_consensus.cc:1261] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Refusing update from remote peer 228187c39a294d7b86852abf364e871c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:01:58.380434 4804 raft_consensus.cc:1261] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Refusing update from remote peer 228187c39a294d7b86852abf364e871c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:01:58.381217 5163 consensus_queue.cc:1035] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:01:58.381400 5161 consensus_queue.cc:1035] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:01:58.389556 5166 mvcc.cc:205] Tried to move back new op lower bound from 7298807064079486976 to 7298807063986720768. Current Snapshot: MvccSnapshot[applied={T|T < 7298807064079486976}]
I20260620 06:01:58.391136 5167 mvcc.cc:205] Tried to move back new op lower bound from 7298807064079486976 to 7298807063986720768. Current Snapshot: MvccSnapshot[applied={T|T < 7298807064079486976}]
I20260620 06:01:58.401858 4646 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:01:58.402098 4646 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:01:58.405201 4646 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: Sending DeleteTablet for 3 replicas of tablet c405ea29160a434889661589c93123ce
I20260620 06:01:58.406215 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet c405ea29160a434889661589c93123ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:01:58 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:37702
I20260620 06:01:58.406215 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet c405ea29160a434889661589c93123ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:01:58 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55000
I20260620 06:01:58.406329 5064 tablet_service.cc:1514] Processing DeleteTablet for tablet c405ea29160a434889661589c93123ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:01:58 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:38140
I20260620 06:01:58.406864 5177 tablet_replica.cc:330] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: stopping tablet replica
I20260620 06:01:58.406962 5178 tablet_replica.cc:330] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: stopping tablet replica
I20260620 06:01:58.407312 5178 raft_consensus.cc:2229] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:01:58.407312 5177 raft_consensus.cc:2229] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:01:58.407701 5178 raft_consensus.cc:2258] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:01:58.407701 5177 raft_consensus.cc:2258] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:01:58.407891 4646 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46436:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:01:58.408311 5176 tablet_replica.cc:330] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: stopping tablet replica
W20260620 06:01:58.408356 4646 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:01:58.409597 5177 ts_tablet_manager.cc:1905] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:01:58.410463 5178 ts_tablet_manager.cc:1905] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:01:58.412895 5177 ts_tablet_manager.cc:1918] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:01:58.413241 5178 ts_tablet_manager.cc:1918] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:01:58.413349 5178 log.cc:1196] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/c405ea29160a434889661589c93123ce
I20260620 06:01:58.413448 5176 raft_consensus.cc:2229] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:01:58.413707 5178 ts_tablet_manager.cc:1939] T c405ea29160a434889661589c93123ce P 6b9edad54daa469caa664940947f1e46: Deleting consensus metadata
I20260620 06:01:58.413967 5176 raft_consensus.cc:2258] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:01:58.414297 4784 tablet_service.cc:1467] Processing CreateTablet for tablet 0da3111d9966411fbc7427235427dfc3 (DEFAULT_TABLE table=test-table [id=436a3c7458704935aaaf6fb6495be2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:58.414604 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da3111d9966411fbc7427235427dfc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.414791 4924 tablet_service.cc:1467] Processing CreateTablet for tablet 0da3111d9966411fbc7427235427dfc3 (DEFAULT_TABLE table=test-table [id=436a3c7458704935aaaf6fb6495be2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:58.414825 5064 tablet_service.cc:1467] Processing CreateTablet for tablet 0da3111d9966411fbc7427235427dfc3 (DEFAULT_TABLE table=test-table [id=436a3c7458704935aaaf6fb6495be2d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:01:58.415062 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da3111d9966411fbc7427235427dfc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.415073 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0da3111d9966411fbc7427235427dfc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.415163 4634 catalog_manager.cc:4841] TS 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783): tablet c405ea29160a434889661589c93123ce (table test-table [id=3e08fb7f203348b88932a3dd3a623423]) successfully deleted
I20260620 06:01:58.417158 5153 tablet_bootstrap.cc:492] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Bootstrap starting.
I20260620 06:01:58.417415 5155 tablet_bootstrap.cc:492] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Bootstrap starting.
I20260620 06:01:58.417963 5154 tablet_bootstrap.cc:492] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Bootstrap starting.
I20260620 06:01:58.418282 5153 tablet_bootstrap.cc:654] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.418351 5155 tablet_bootstrap.cc:654] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.418955 5154 tablet_bootstrap.cc:654] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.414283 5177 log.cc:1196] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/c405ea29160a434889661589c93123ce
I20260620 06:01:58.419562 5177 ts_tablet_manager.cc:1939] T c405ea29160a434889661589c93123ce P f08355d8734a45d3918c8bf6c6f68892: Deleting consensus metadata
I20260620 06:01:58.420886 4633 catalog_manager.cc:4841] TS f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843): tablet c405ea29160a434889661589c93123ce (table test-table [id=3e08fb7f203348b88932a3dd3a623423]) successfully deleted
I20260620 06:01:58.420939 5176 ts_tablet_manager.cc:1905] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:01:58.423656 5176 ts_tablet_manager.cc:1918] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:01:58.423789 5176 log.cc:1196] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/c405ea29160a434889661589c93123ce
I20260620 06:01:58.424105 5176 ts_tablet_manager.cc:1939] T c405ea29160a434889661589c93123ce P 228187c39a294d7b86852abf364e871c: Deleting consensus metadata
I20260620 06:01:58.425400 4634 catalog_manager.cc:4841] TS 228187c39a294d7b86852abf364e871c (127.3.115.2:34977): tablet c405ea29160a434889661589c93123ce (table test-table [id=3e08fb7f203348b88932a3dd3a623423]) successfully deleted
I20260620 06:01:58.425649 5153 tablet_bootstrap.cc:492] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: No bootstrap required, opened a new log
I20260620 06:01:58.425751 5153 ts_tablet_manager.cc:1397] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20260620 06:01:58.425957 5155 tablet_bootstrap.cc:492] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: No bootstrap required, opened a new log
I20260620 06:01:58.426029 5155 ts_tablet_manager.cc:1397] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260620 06:01:58.426179 5154 tablet_bootstrap.cc:492] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: No bootstrap required, opened a new log
I20260620 06:01:58.426263 5154 ts_tablet_manager.cc:1397] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260620 06:01:58.426301 5153 raft_consensus.cc:348] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.426424 5153 raft_consensus.cc:374] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.426457 5153 raft_consensus.cc:729] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08355d8734a45d3918c8bf6c6f68892, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.426447 5155 raft_consensus.cc:348] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.426549 5155 raft_consensus.cc:374] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.426578 5155 raft_consensus.cc:729] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228187c39a294d7b86852abf364e871c, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.426597 5153 consensus_queue.cc:260] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.426692 5155 consensus_queue.cc:260] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.426774 5154 raft_consensus.cc:348] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.426882 5154 raft_consensus.cc:374] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.426872 5153 ts_tablet_manager.cc:1428] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:01:58.426920 5154 raft_consensus.cc:729] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9edad54daa469caa664940947f1e46, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.426939 5155 ts_tablet_manager.cc:1428] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:01:58.427045 5154 consensus_queue.cc:260] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.427291 5154 ts_tablet_manager.cc:1428] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260620 06:01:58.477504 4991 tablet.cc:2369] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:01:58.485354 5131 tablet.cc:2369] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:01:58.505646 4851 tablet.cc:2369] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:01:58.745604 5159 raft_consensus.cc:482] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:01:58.745767 5159 raft_consensus.cc:504] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.746994 5159 leader_election.cc:288] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843), 228187c39a294d7b86852abf364e871c (127.3.115.2:34977)
I20260620 06:01:58.751101 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.747399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:57541 (local address 127.3.115.2:34977)
0620 06:01:58.747615 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.747619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.747678 (+ 59us) server_negotiation.cc:407] Connection header received
0620 06:01:58.747864 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.747870 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.747927 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.748002 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.748349 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.748942 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.750121 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.750505 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.750587 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.750674 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.750881 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.750967 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260620 06:01:58.751168 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.747367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:47653 (local address 127.3.115.1:35843)
0620 06:01:58.747509 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.747514 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.747797 (+ 283us) server_negotiation.cc:407] Connection header received
0620 06:01:58.747935 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.747938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.747998 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.748081 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.748386 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.749007 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.750076 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.750673 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.750776 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.750869 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.750985 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.751053 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:01:58.751185 5182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.747269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35843 (local address 127.3.115.3:47653)
0620 06:01:58.747675 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.747691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.747831 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.748111 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.748114 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.748129 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.748252 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.748257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.749118 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.749120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.749956 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.749964 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.750783 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.750786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.750888 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.750905 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.750961 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.751012 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":56,"thread_start_us":76,"threads_started":1}
I20260620 06:01:58.751574 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.747263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:34977 (local address 127.3.115.3:57541)
0620 06:01:58.747612 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.747624 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.747724 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.748036 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.748042 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.748077 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.748234 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.748240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.749050 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.749055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.749979 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.749988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.750618 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.750622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.750728 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.750759 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.751320 (+ 561us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.751409 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":85,"thread_start_us":97,"threads_started":1}
I20260620 06:01:58.751677 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da3111d9966411fbc7427235427dfc3" candidate_uuid: "6b9edad54daa469caa664940947f1e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08355d8734a45d3918c8bf6c6f68892" is_pre_election: true
I20260620 06:01:58.751833 4804 raft_consensus.cc:2454] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9edad54daa469caa664940947f1e46 in term 0.
I20260620 06:01:58.752090 4944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da3111d9966411fbc7427235427dfc3" candidate_uuid: "6b9edad54daa469caa664940947f1e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228187c39a294d7b86852abf364e871c" is_pre_election: true
I20260620 06:01:58.752239 5017 leader_election.cc:302] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9edad54daa469caa664940947f1e46, f08355d8734a45d3918c8bf6c6f68892; no voters:
I20260620 06:01:58.752341 4944 raft_consensus.cc:2454] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b9edad54daa469caa664940947f1e46 in term 0.
I20260620 06:01:58.752595 5159 raft_consensus.cc:2781] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:01:58.752643 5159 raft_consensus.cc:482] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:01:58.752707 5159 raft_consensus.cc:3037] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.753994 5159 raft_consensus.cc:504] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.754379 5159 leader_election.cc:288] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [CANDIDATE]: Term 1 election: Requested vote from peers f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843), 228187c39a294d7b86852abf364e871c (127.3.115.2:34977)
I20260620 06:01:58.754733 4944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da3111d9966411fbc7427235427dfc3" candidate_uuid: "6b9edad54daa469caa664940947f1e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228187c39a294d7b86852abf364e871c"
I20260620 06:01:58.754887 4944 raft_consensus.cc:3037] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.754890 4804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0da3111d9966411fbc7427235427dfc3" candidate_uuid: "6b9edad54daa469caa664940947f1e46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f08355d8734a45d3918c8bf6c6f68892"
I20260620 06:01:58.755007 4804 raft_consensus.cc:3037] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.756091 4944 raft_consensus.cc:2454] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9edad54daa469caa664940947f1e46 in term 1.
I20260620 06:01:58.756091 4804 raft_consensus.cc:2454] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b9edad54daa469caa664940947f1e46 in term 1.
I20260620 06:01:58.756413 5017 leader_election.cc:302] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9edad54daa469caa664940947f1e46, f08355d8734a45d3918c8bf6c6f68892; no voters:
I20260620 06:01:58.756641 5159 raft_consensus.cc:2781] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:58.756846 5159 raft_consensus.cc:686] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 1 LEADER]: Becoming Leader. State: Replica: 6b9edad54daa469caa664940947f1e46, State: Running, Role: LEADER
I20260620 06:01:58.757079 5159 consensus_queue.cc:237] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.759516 4647 catalog_manager.cc:5495] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b9edad54daa469caa664940947f1e46 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "6b9edad54daa469caa664940947f1e46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } health_report { overall_health: HEALTHY } } }
Jun 20 06:01:58 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935317, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:01:58.810983 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.802773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36633 (local address 127.0.0.1:46446)
0620 06:01:58.803044 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.803057 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.803189 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.803454 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.803456 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.803921 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:01:58.804010 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.804016 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.804782 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.804784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.805344 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.805351 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.805796 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.805798 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.805877 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.806433 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:01:58.806444 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:01:58.807897 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:01:58.809511 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:58.809519 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:58.809525 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:58.809745 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:58.809988 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:01:58.809990 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:01:58.809991 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:01:58.810073 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:01:58.810458 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:01:58.810464 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:01:58.810565 (+ 101us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.810679 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.810848 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260620 06:01:58.811075 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.802887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46446 (local address 127.3.115.62:36633)
0620 06:01:58.803102 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.803105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.803120 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:01:58.803283 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.803285 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.803326 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.803441 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:58.804111 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.804641 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.805463 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.805684 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.805797 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.808024 (+ 2227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:58.808028 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:58.808030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:58.808050 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:58.809415 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.809830 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.809833 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.809834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.809909 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:58.810148 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:58.810151 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:58.810152 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:58.810316 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:58.810415 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.810732 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.810931 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260620 06:01:58.814920 4647 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:46446:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:01:58.815675 4647 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:01:58.820652 4784 tablet_service.cc:1467] Processing CreateTablet for tablet b13b0f5e6372453cb4b8993fcbeda538 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e21b996e78a44058cb8a09186d535da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:01:58.820761 5064 tablet_service.cc:1467] Processing CreateTablet for tablet b13b0f5e6372453cb4b8993fcbeda538 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e21b996e78a44058cb8a09186d535da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:01:58.820761 4924 tablet_service.cc:1467] Processing CreateTablet for tablet b13b0f5e6372453cb4b8993fcbeda538 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1e21b996e78a44058cb8a09186d535da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:01:58.821118 4924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13b0f5e6372453cb4b8993fcbeda538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.821118 5064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13b0f5e6372453cb4b8993fcbeda538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.821120 4784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b13b0f5e6372453cb4b8993fcbeda538. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:01:58.823628 5153 tablet_bootstrap.cc:492] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: Bootstrap starting.
I20260620 06:01:58.823824 5155 tablet_bootstrap.cc:492] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: Bootstrap starting.
I20260620 06:01:58.823890 5154 tablet_bootstrap.cc:492] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: Bootstrap starting.
I20260620 06:01:58.824457 5153 tablet_bootstrap.cc:654] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.824764 5155 tablet_bootstrap.cc:654] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.824764 5154 tablet_bootstrap.cc:654] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: Neither blocks nor log segments found. Creating new log.
I20260620 06:01:58.826000 5155 tablet_bootstrap.cc:492] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: No bootstrap required, opened a new log
I20260620 06:01:58.826033 5153 tablet_bootstrap.cc:492] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: No bootstrap required, opened a new log
I20260620 06:01:58.826084 5155 ts_tablet_manager.cc:1397] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:01:58.826114 5153 ts_tablet_manager.cc:1397] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:01:58.826123 5154 tablet_bootstrap.cc:492] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: No bootstrap required, opened a new log
I20260620 06:01:58.826188 5154 ts_tablet_manager.cc:1397] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:01:58.826519 5155 raft_consensus.cc:348] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.826655 5155 raft_consensus.cc:374] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.826560 5154 raft_consensus.cc:348] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.826556 5153 raft_consensus.cc:348] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.826696 5155 raft_consensus.cc:729] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 228187c39a294d7b86852abf364e871c, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.826711 5154 raft_consensus.cc:374] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.826718 5153 raft_consensus.cc:374] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:01:58.826746 5154 raft_consensus.cc:729] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b9edad54daa469caa664940947f1e46, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.826761 5153 raft_consensus.cc:729] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f08355d8734a45d3918c8bf6c6f68892, State: Initialized, Role: FOLLOWER
I20260620 06:01:58.826828 5155 consensus_queue.cc:260] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.826869 5154 consensus_queue.cc:260] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.826881 5153 consensus_queue.cc:260] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.827106 5155 ts_tablet_manager.cc:1428] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:01:58.827106 5154 ts_tablet_manager.cc:1428] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:01:58.827205 5153 ts_tablet_manager.cc:1428] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:01:58.827139 5160 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.827280 5160 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.827334 5163 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.827414 5159 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.827493 5163 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.827546 5159 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.880077 5160 raft_consensus.cc:482] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:01:58.880223 5160 raft_consensus.cc:504] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.881462 5160 leader_election.cc:288] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 228187c39a294d7b86852abf364e871c (127.3.115.2:34977), 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783)
I20260620 06:01:58.886255 5151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.881805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:42919 (local address 127.3.115.2:34977)
0620 06:01:58.881968 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.881972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.882151 (+ 179us) server_negotiation.cc:407] Connection header received
0620 06:01:58.882410 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.882415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.882469 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.882526 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.883029 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.883755 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.885076 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.885667 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.885748 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.885799 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.886032 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.886116 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 06:01:58.886286 5197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.882176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:42783 (local address 127.3.115.1:42597)
0620 06:01:58.882621 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.882633 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.882724 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.883029 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.883032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.883050 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.883196 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.883203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.884109 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.884111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.885212 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.885222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.885872 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.885876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.885966 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.886011 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.886087 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.886146 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":113,"threads_started":1}
I20260620 06:01:58.886389 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.882273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:42597 (local address 127.3.115.3:42783)
0620 06:01:58.882640 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.882645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.882687 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:01:58.882821 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.882824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.882881 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.882969 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:01:58.883373 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.883980 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.885318 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.885755 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.885886 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.885950 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.886115 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.886237 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":57,"threads_started":1}
I20260620 06:01:58.886268 5196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.881702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:34977 (local address 127.3.115.1:42919)
0620 06:01:58.882065 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.882080 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.882226 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.882675 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.882679 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.882703 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:01:58.882845 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.882852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.883866 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.883870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.884964 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.884973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.885785 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.885791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.885900 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.885924 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.886008 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.886084 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":150,"threads_started":1}
I20260620 06:01:58.886941 5084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13b0f5e6372453cb4b8993fcbeda538" candidate_uuid: "f08355d8734a45d3918c8bf6c6f68892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9edad54daa469caa664940947f1e46" is_pre_election: true
I20260620 06:01:58.887063 5084 raft_consensus.cc:2454] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f08355d8734a45d3918c8bf6c6f68892 in term 0.
I20260620 06:01:58.887195 4944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13b0f5e6372453cb4b8993fcbeda538" candidate_uuid: "f08355d8734a45d3918c8bf6c6f68892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228187c39a294d7b86852abf364e871c" is_pre_election: true
I20260620 06:01:58.887354 4944 raft_consensus.cc:2454] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f08355d8734a45d3918c8bf6c6f68892 in term 0.
I20260620 06:01:58.887475 4738 leader_election.cc:302] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9edad54daa469caa664940947f1e46, f08355d8734a45d3918c8bf6c6f68892; no voters:
I20260620 06:01:58.887758 5160 raft_consensus.cc:2781] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:01:58.887811 5160 raft_consensus.cc:482] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:01:58.887837 5160 raft_consensus.cc:3037] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.888839 5160 raft_consensus.cc:504] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.889182 5160 leader_election.cc:288] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [CANDIDATE]: Term 1 election: Requested vote from peers 228187c39a294d7b86852abf364e871c (127.3.115.2:34977), 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783)
I20260620 06:01:58.889541 4944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13b0f5e6372453cb4b8993fcbeda538" candidate_uuid: "f08355d8734a45d3918c8bf6c6f68892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "228187c39a294d7b86852abf364e871c"
I20260620 06:01:58.889679 4944 raft_consensus.cc:3037] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.889660 5084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b13b0f5e6372453cb4b8993fcbeda538" candidate_uuid: "f08355d8734a45d3918c8bf6c6f68892" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b9edad54daa469caa664940947f1e46"
I20260620 06:01:58.889775 5084 raft_consensus.cc:3037] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:01:58.890836 4944 raft_consensus.cc:2454] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f08355d8734a45d3918c8bf6c6f68892 in term 1.
I20260620 06:01:58.890829 5084 raft_consensus.cc:2454] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f08355d8734a45d3918c8bf6c6f68892 in term 1.
I20260620 06:01:58.891108 4738 leader_election.cc:302] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b9edad54daa469caa664940947f1e46, f08355d8734a45d3918c8bf6c6f68892; no voters:
I20260620 06:01:58.891294 5160 raft_consensus.cc:2781] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:01:58.891479 5160 raft_consensus.cc:686] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [term 1 LEADER]: Becoming Leader. State: Replica: f08355d8734a45d3918c8bf6c6f68892, State: Running, Role: LEADER
I20260620 06:01:58.891763 5160 consensus_queue.cc:237] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } }
I20260620 06:01:58.893039 5199 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: TxnStatusTablet state changed. Reason: New leader f08355d8734a45d3918c8bf6c6f68892. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.893159 5199 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:01:58.893589 5201 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:01:58.893774 5201 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:01:58.894309 4645 catalog_manager.cc:5495] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 reported cstate change: term changed from 0 to 1, leader changed from <none> to f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } health_report { overall_health: UNKNOWN } } }
I20260620 06:01:58.907954 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.904143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55004 (local address 127.3.115.1:35843)
0620 06:01:58.904295 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.904299 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.904452 (+ 153us) server_negotiation.cc:407] Connection header received
0620 06:01:58.904722 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.904725 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.904783 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.904894 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:01:58.905132 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.905652 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.906496 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.906716 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.906835 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.907150 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.907560 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.907679 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.907758 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.907812 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:01:58.907954 5191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.904212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35843 (local address 127.0.0.1:55004)
0620 06:01:58.904367 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.904388 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.904520 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.904904 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.904907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.904917 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:01:58.905016 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.905021 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.905761 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.905767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.906380 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.906388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.906832 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.906836 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.906948 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.906997 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.907666 (+ 669us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.907682 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.907749 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.907833 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:01:58.911334 5084 raft_consensus.cc:1261] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Refusing update from remote peer f08355d8734a45d3918c8bf6c6f68892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:01:58.911464 4944 raft_consensus.cc:1261] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Refusing update from remote peer f08355d8734a45d3918c8bf6c6f68892: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:01:58.911890 5160 consensus_queue.cc:1035] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:01:58.912081 5199 consensus_queue.cc:1035] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892 [LEADER]: Connected to new peer: Peer: permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:01:58.913527 5159 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: TxnStatusTablet state changed. Reason: New leader f08355d8734a45d3918c8bf6c6f68892. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.913650 5159 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.913846 5163 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: TxnStatusTablet state changed. Reason: New leader f08355d8734a45d3918c8bf6c6f68892. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.913971 5163 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.916570 5160 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.916709 5160 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:01:58.917407 5160 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.917515 5160 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P f08355d8734a45d3918c8bf6c6f68892: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:01:58.917809 5204 mvcc.cc:205] Tried to move back new op lower bound from 7298807066255380480 to 7298807066183135232. Current Snapshot: MvccSnapshot[applied={T|T < 7298807066255380480 or (T in {7298807066255380480})}]
I20260620 06:01:58.918715 5163 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.918830 5163 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P 228187c39a294d7b86852abf364e871c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.918776 5159 tablet_replica.cc:439] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f08355d8734a45d3918c8bf6c6f68892" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 } } peers { permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 } } peers { permanent_uuid: "6b9edad54daa469caa664940947f1e46" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 42783 } } }
I20260620 06:01:58.918895 5159 tablet_replica.cc:442] T b13b0f5e6372453cb4b8993fcbeda538 P 6b9edad54daa469caa664940947f1e46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:01:58.927711 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.923153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38144 (local address 127.3.115.3:42783)
0620 06:01:58.923298 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.923302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.923425 (+ 123us) server_negotiation.cc:407] Connection header received
0620 06:01:58.923620 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.923623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.923692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.923811 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:01:58.924112 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.924926 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.926099 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.926351 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.926458 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.926743 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.927319 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.927458 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.927505 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.927561 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:01:58.938930 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.935162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:50047 (local address 127.3.115.1:35843)
0620 06:01:58.935286 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.935291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.935427 (+ 136us) server_negotiation.cc:407] Connection header received
0620 06:01:58.935534 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.935537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.935589 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.935671 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:01:58.935884 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.936680 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.937591 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.937939 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.938040 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.938452 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.938525 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.938610 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.938726 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.938774 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:01:58.938942 5216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.935059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35843 (local address 127.3.115.3:50047)
0620 06:01:58.935338 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.935357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.935444 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.935684 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.935686 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.935692 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:01:58.935788 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.935792 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.936782 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.936784 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.937484 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.937493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.938058 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.938060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.938179 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.938255 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.938616 (+ 361us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.938624 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.938713 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.938792 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":109,"threads_started":1}
I20260620 06:01:58.946033 4944 raft_consensus.cc:1261] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Refusing update from remote peer 6b9edad54daa469caa664940947f1e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:01:58.946095 4804 raft_consensus.cc:1261] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Refusing update from remote peer 6b9edad54daa469caa664940947f1e46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:01:58.946559 5184 consensus_queue.cc:1035] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f08355d8734a45d3918c8bf6c6f68892" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:01:58.946741 5159 consensus_queue.cc:1035] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "228187c39a294d7b86852abf364e871c" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 34977 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:01:58.970078 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.966044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:54147 (local address 127.3.115.3:42783)
0620 06:01:58.966154 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:58.966158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:58.966395 (+ 237us) server_negotiation.cc:407] Connection header received
0620 06:01:58.966482 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:58.966485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:58.966532 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:58.966610 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:01:58.966833 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.967623 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.968970 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.969180 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.969263 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.969629 (+ 366us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.969694 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.969749 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:58.969871 (+ 122us) server_negotiation.cc:299] Negotiation successful
0620 06:01:58.969913 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:01:58.970095 5220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:58.965980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:42783 (local address 127.3.115.1:54147)
0620 06:01:58.966309 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0620 06:01:58.966322 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:01:58.966409 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:01:58.966627 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:01:58.966628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:01:58.966634 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:01:58.966738 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.966742 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.967753 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.967757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.968827 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:01:58.968839 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:01:58.969311 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:01:58.969315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:01:58.969447 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:58.969503 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:01:58.969776 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:01:58.969782 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:01:58.969849 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:01:58.969925 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":123,"threads_started":1}
I20260620 06:01:58.998023 4645 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:01:58.998201 4645 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46436:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:01:59.000701 4645 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98ed1d82c2a441d09d45d7b841653b33: Sending DeleteTablet for 3 replicas of tablet 0da3111d9966411fbc7427235427dfc3
I20260620 06:01:59.001266 4784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0da3111d9966411fbc7427235427dfc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:01:58 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55000
I20260620 06:01:59.001636 4924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0da3111d9966411fbc7427235427dfc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:01:58 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:37702
I20260620 06:01:59.001901 5064 tablet_service.cc:1514] Processing DeleteTablet for tablet 0da3111d9966411fbc7427235427dfc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:01:58 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:38140
I20260620 06:01:59.003423 5229 tablet_replica.cc:330] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: stopping tablet replica
I20260620 06:01:59.003558 5229 raft_consensus.cc:2229] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:01:59.003883 5229 raft_consensus.cc:2258] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:01:59.004115 5228 tablet_replica.cc:330] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: stopping tablet replica
I20260620 06:01:59.004244 5228 raft_consensus.cc:2229] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:01:59.004382 5228 raft_consensus.cc:2258] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:01:59.004606 5229 ts_tablet_manager.cc:1905] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:01:59.005034 5228 ts_tablet_manager.cc:1905] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:01:59.007645 5229 ts_tablet_manager.cc:1918] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:01:59.007738 5229 log.cc:1196] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/0da3111d9966411fbc7427235427dfc3
I20260620 06:01:59.007915 5228 ts_tablet_manager.cc:1918] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:01:59.008004 5228 log.cc:1196] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/0da3111d9966411fbc7427235427dfc3
I20260620 06:01:59.008042 5229 ts_tablet_manager.cc:1939] T 0da3111d9966411fbc7427235427dfc3 P 6b9edad54daa469caa664940947f1e46: Deleting consensus metadata
I20260620 06:01:59.008113 5227 tablet_replica.cc:330] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: stopping tablet replica
I20260620 06:01:59.008247 5228 ts_tablet_manager.cc:1939] T 0da3111d9966411fbc7427235427dfc3 P 228187c39a294d7b86852abf364e871c: Deleting consensus metadata
I20260620 06:01:59.008340 5227 raft_consensus.cc:2229] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:01:59.008524 5227 raft_consensus.cc:2258] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:01:59.009034 4634 catalog_manager.cc:4841] TS 228187c39a294d7b86852abf364e871c (127.3.115.2:34977): tablet 0da3111d9966411fbc7427235427dfc3 (table test-table [id=436a3c7458704935aaaf6fb6495be2d5]) successfully deleted
I20260620 06:01:59.009235 4634 catalog_manager.cc:4841] TS 6b9edad54daa469caa664940947f1e46 (127.3.115.3:42783): tablet 0da3111d9966411fbc7427235427dfc3 (table test-table [id=436a3c7458704935aaaf6fb6495be2d5]) successfully deleted
I20260620 06:01:59.013183 5227 ts_tablet_manager.cc:1905] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:01:59.016258 5227 ts_tablet_manager.cc:1918] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:01:59.016345 5227 log.cc:1196] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/0da3111d9966411fbc7427235427dfc3
I20260620 06:01:59.016628 5227 ts_tablet_manager.cc:1939] T 0da3111d9966411fbc7427235427dfc3 P f08355d8734a45d3918c8bf6c6f68892: Deleting consensus metadata
I20260620 06:01:59.017524 4633 catalog_manager.cc:4841] TS f08355d8734a45d3918c8bf6c6f68892 (127.3.115.1:35843): tablet 0da3111d9966411fbc7427235427dfc3 (table test-table [id=436a3c7458704935aaaf6fb6495be2d5]) successfully deleted
Jun 20 06:01:59 dist-test-slave-4h14 krb5kdc[4602](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:01:59.022423 5149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:59.010853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55016 (local address 127.3.115.1:35843)
0620 06:01:59.010981 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:59.010987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:59.011332 (+ 345us) server_negotiation.cc:407] Connection header received
0620 06:01:59.011501 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:59.011504 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:59.011551 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:59.011652 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:59.012321 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:59.012969 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:59.014133 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:59.014473 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:59.014547 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:59.018731 (+ 4184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:59.018741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:59.018760 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:59.018785 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:59.020525 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:59.021080 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:59.021091 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:59.021092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:59.021171 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:59.021459 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:59.021464 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:59.021466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:59.021723 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:59.021950 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:59.022107 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 06:01:59.022229 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 06:01:59.022907 4844 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55016
I20260620 06:01:59.032698 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:01:59.024736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46454 (local address 127.3.115.62:36633)
0620 06:01:59.024888 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:01:59.024894 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:01:59.025031 (+ 137us) server_negotiation.cc:407] Connection header received
0620 06:01:59.025172 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:01:59.025175 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:01:59.025229 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:01:59.025317 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:01:59.025949 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:59.026483 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:59.027437 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:01:59.027737 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:01:59.027855 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:01:59.029450 (+ 1595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:01:59.029454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:01:59.029457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:01:59.029482 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:01:59.030996 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:59.031430 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:59.031435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:59.031437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:59.031518 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:01:59.031789 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:01:59.031794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:01:59.031795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:01:59.031978 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:01:59.032119 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:01:59.032353 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 06:01:59.032478 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 06:01:59.033097 4645 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46454
I20260620 06:01:59.034523 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4719
I20260620 06:01:59.042712 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4859
I20260620 06:01:59.049621 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4999
I20260620 06:01:59.055621 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 4615
2026-06-20T06:01:59Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2929 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:01:59 dist-test-slave-4h14 krb5kdc[5236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:01:59 dist-test-slave-4h14 krb5kdc[5236](info): set up 2 sockets
Jun 20 06:01:59 dist-test-slave-4h14 krb5kdc[5236](info): commencing operation
krb5kdc: starting...
W20260620 06:02:00.101207 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:00.227054 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34993
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:34993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:00.339445 5249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:00.339744 5249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:00.339787 5249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:00.343978 5249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:00.344051 5249 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:02:00.344069 5249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:00.344087 5249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:00.344105 5249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:00.344120 5249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:00.349756 5249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:34993
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:00.350929 5249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:00.352064 5249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:00.360661 5255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:00.360730 5254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:00.360678 5257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:00.361007 5249 server_base.cc:1034] running on GCE node
I20260620 06:02:00.361666 5249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:00.362830 5249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:00.364017 5249 hybrid_clock.cc:648] HybridClock initialized: now 1781935320363985 us; error 47 us; skew 500 ppm
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:00.366870 5249 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:00.367774 5249 webserver.cc:460] Webserver started at http://127.3.115.62:46577/ using document root <none> and password file <none>
I20260620 06:02:00.368384 5249 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:00.368476 5249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:00.368767 5249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:00.370842 5249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "1e6f40964de4480c93fd18eb0f134f0a"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "9ac083ebb98e2a525b97e1634ff60d87"
server_key_iv: "a7e0d7bea0e7d8cb386be138aee03d60"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.371421 5249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "1e6f40964de4480c93fd18eb0f134f0a"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "9ac083ebb98e2a525b97e1634ff60d87"
server_key_iv: "a7e0d7bea0e7d8cb386be138aee03d60"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.375155 5249 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:02:00.377723 5264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:00.379103 5249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:00.379249 5249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "1e6f40964de4480c93fd18eb0f134f0a"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "9ac083ebb98e2a525b97e1634ff60d87"
server_key_iv: "a7e0d7bea0e7d8cb386be138aee03d60"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.379390 5249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:00.394668 5249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:00.398253 5249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:00.398480 5249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:00.407270 5249 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:34993
I20260620 06:02:00.407294 5326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:34993 every 8 connection(s)
I20260620 06:02:00.408340 5249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:00.411314 5327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:00.413615 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 5249
I20260620 06:02:00.413710 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:00.414016 3532 external_mini_cluster.cc:1428] Setting key b0eaa9c193a4007871bdcb4965dc27ad
I20260620 06:02:00.417672 5327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Bootstrap starting.
I20260620 06:02:00.420024 5327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:00.420884 5327 log.cc:824] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:00.423226 5327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: No bootstrap required, opened a new log
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:00.427021 5327 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6f40964de4480c93fd18eb0f134f0a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34993 } }
I20260620 06:02:00.427277 5327 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:00.427351 5327 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e6f40964de4480c93fd18eb0f134f0a, State: Initialized, Role: FOLLOWER
I20260620 06:02:00.427892 5327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6f40964de4480c93fd18eb0f134f0a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34993 } }
I20260620 06:02:00.428056 5327 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:00.428133 5327 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:00.428232 5327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:00.429234 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.415575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.3.115.62:34993)
0620 06:02:00.416037 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:00.416052 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:00.416101 (+ 49us) server_negotiation.cc:407] Connection header received
0620 06:02:00.416867 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:00.416887 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:00.417234 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:00.417600 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:00.418399 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.419451 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.420368 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.420920 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.421098 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.423876 (+ 2778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:00.423882 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:00.423897 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:00.423925 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:00.425828 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.426344 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.426349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.426354 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.426473 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.426781 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.426786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.426788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.427247 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:00.427467 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:00.427840 (+ 373us) server_negotiation.cc:299] Negotiation successful
0620 06:02:00.428168 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":141,"threads_started":1}
I20260620 06:02:00.429452 5327 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6f40964de4480c93fd18eb0f134f0a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34993 } }
I20260620 06:02:00.429827 5327 leader_election.cc:302] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e6f40964de4480c93fd18eb0f134f0a; no voters:
I20260620 06:02:00.430160 5327 leader_election.cc:288] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:00.430353 5332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:00.430639 5332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [term 1 LEADER]: Becoming Leader. State: Replica: 1e6f40964de4480c93fd18eb0f134f0a, State: Running, Role: LEADER
I20260620 06:02:00.431007 5332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6f40964de4480c93fd18eb0f134f0a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34993 } }
I20260620 06:02:00.431408 5327 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:00.432964 5334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e6f40964de4480c93fd18eb0f134f0a. Latest consensus state: current_term: 1 leader_uuid: "1e6f40964de4480c93fd18eb0f134f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6f40964de4480c93fd18eb0f134f0a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34993 } } }
I20260620 06:02:00.433000 5333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e6f40964de4480c93fd18eb0f134f0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e6f40964de4480c93fd18eb0f134f0a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34993 } } }
I20260620 06:02:00.433089 5334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:00.433100 5333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:00.433583 5342 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:00.436638 5342 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:00.442734 5342 catalog_manager.cc:1347] Generated new cluster ID: 6abec782adf54d0fae19eca26d8b79cb
I20260620 06:02:00.442823 5342 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:00.454347 5342 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:00.455626 5342 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:00.461685 5342 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Generated new TSK 0
I20260620 06:02:00.462409 5342 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:00.496482 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34993
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:00.625274 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:00.625560 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:00.625610 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:00.629721 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:00.629791 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:00.629868 5353 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:02:00.629899 5353 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:02:00.629957 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:00.635618 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:34993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:00.636771 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:00.637861 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:00.647761 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:00.647765 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:00.647761 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:00.648132 5353 server_base.cc:1034] running on GCE node
I20260620 06:02:00.648615 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:00.649444 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:00.650619 5353 hybrid_clock.cc:648] HybridClock initialized: now 1781935320650602 us; error 32 us; skew 500 ppm
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:00.653998 5353 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:00.655016 5353 webserver.cc:460] Webserver started at http://127.3.115.1:38785/ using document root <none> and password file <none>
I20260620 06:02:00.655616 5353 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:00.655681 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:00.655915 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:00.658155 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "edfeac70e44e4bf0871fff45cf2eb464"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "82e9a65b91e2e6e0d4c61d378f6793b2"
server_key_iv: "45ae1c4b03d58001ee52f92e1f934fba"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.658751 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "edfeac70e44e4bf0871fff45cf2eb464"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "82e9a65b91e2e6e0d4c61d378f6793b2"
server_key_iv: "45ae1c4b03d58001ee52f92e1f934fba"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.662828 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:00.665613 5368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:00.667024 5353 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:00.667183 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "edfeac70e44e4bf0871fff45cf2eb464"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "82e9a65b91e2e6e0d4c61d378f6793b2"
server_key_iv: "45ae1c4b03d58001ee52f92e1f934fba"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.667313 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:00.680887 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:00.684350 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:00.684559 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:00.686026 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:00.686113 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:00.686182 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:00.686232 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:00.696974 5353 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35353
I20260620 06:02:00.697060 5483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35353 every 8 connection(s)
I20260620 06:02:00.698056 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:00.701368 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.687032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37673 (local address 127.3.115.62:34993)
0620 06:02:00.687163 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:00.687167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:00.688283 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 06:02:00.689542 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:00.689546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:00.689606 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:00.689700 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:00.691439 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.692319 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.693879 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.694175 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.694295 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.697560 (+ 3265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:00.697564 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:00.697566 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:00.697593 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:00.699275 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.699751 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.699754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.699755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.699830 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.700233 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.700236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.700237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.700430 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:00.700522 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:00.701131 (+ 609us) server_negotiation.cc:299] Negotiation successful
0620 06:02:00.701245 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:02:00.702178 5377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.687628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.3.115.1:37673)
0620 06:02:00.688127 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:00.688173 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:00.689289 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:00.689920 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:00.689935 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:00.690449 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:00.691250 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.691274 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.692482 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.692487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.693715 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.693730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.694335 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.694340 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.694532 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.695257 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:00.695281 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:00.697349 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:00.699381 (+ 2032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.699386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.699397 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.699651 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.699955 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.699957 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.699959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.700101 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.700551 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:00.700557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:00.700890 (+ 333us) client_negotiation.cc:769] Sending connection context
0620 06:02:00.701052 (+ 162us) client_negotiation.cc:240] Negotiation successful
0620 06:02:00.701311 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":137,"threads_started":1}
I20260620 06:02:00.703627 5484 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34993
I20260620 06:02:00.704039 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 5353
I20260620 06:02:00.704154 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:00.704419 3532 external_mini_cluster.cc:1428] Setting key a8c38c71bbc8cccafeec371da54db998
I20260620 06:02:00.704841 5484 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:00.705371 5484 heartbeater.cc:511] Master 127.3.115.62:34993 requested a full tablet report, sending...
I20260620 06:02:00.707022 5281 ts_manager.cc:194] Registered new tserver with Master: edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353)
I20260620 06:02:00.708515 5281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:37673
I20260620 06:02:00.714852 5490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.706298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.3.115.1:51675)
0620 06:02:00.706574 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:00.706589 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:00.706695 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:00.706944 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:00.706947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:00.707169 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:00.707295 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.707303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.708550 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.708553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.709519 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.709527 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.710143 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.710147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.710236 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.710832 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:00.710846 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:00.711275 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:00.713425 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.713427 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.713429 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.713619 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.713919 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.713921 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.713922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.713995 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.714441 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:00.714443 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:00.714515 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 06:02:00.714595 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:02:00.714702 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":101,"threads_started":1}
I20260620 06:02:00.714982 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.706050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:51675 (local address 127.3.115.62:34993)
0620 06:02:00.706186 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:00.706191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:00.706673 (+ 482us) server_negotiation.cc:407] Connection header received
0620 06:02:00.706778 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:00.706781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:00.706824 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:00.706907 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:00.707413 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.708394 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.709635 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.710014 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.710136 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.711416 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:00.711422 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:00.711428 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:00.711460 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:00.713306 (+ 1846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.713721 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.713725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.713727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.713821 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.714087 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.714090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.714092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.714341 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:00.714455 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:00.714684 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 06:02:00.714800 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":214,"server-negotiator.queue_time_us":58}
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:00.735931 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34993
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:00.853269 5493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:00.853611 5493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:00.853665 5493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:00.858222 5493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:00.858307 5493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:00.858352 5493 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:02:00.858368 5493 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:02:00.858460 5493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:00.864238 5493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:34993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:00.865500 5493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:00.866590 5493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:00.876996 5499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:00.877003 5498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:00.876996 5501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:00.877698 5493 server_base.cc:1034] running on GCE node
I20260620 06:02:00.878157 5493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:00.878881 5493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:00.880060 5493 hybrid_clock.cc:648] HybridClock initialized: now 1781935320880044 us; error 52 us; skew 500 ppm
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:00.883265 5493 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:00.884379 5493 webserver.cc:460] Webserver started at http://127.3.115.2:45641/ using document root <none> and password file <none>
I20260620 06:02:00.885064 5493 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:00.885136 5493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:00.885370 5493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:00.887601 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "847d2979047844d7a9bbbaee2165e163"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "765b1bdb56f68486e9320614c544c914"
server_key_iv: "2c6f4d2e825d5e35fad9e764c241025f"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.888235 5493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "847d2979047844d7a9bbbaee2165e163"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "765b1bdb56f68486e9320614c544c914"
server_key_iv: "2c6f4d2e825d5e35fad9e764c241025f"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.892540 5493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:00.895521 5508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:00.897083 5493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:00.897226 5493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "847d2979047844d7a9bbbaee2165e163"
format_stamp: "Formatted at 2026-06-20 06:02:00 on dist-test-slave-4h14"
server_key: "765b1bdb56f68486e9320614c544c914"
server_key_iv: "2c6f4d2e825d5e35fad9e764c241025f"
server_key_version: "encryptionkey@0"
I20260620 06:02:00.897363 5493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:00.926589 5493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:00.930335 5493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:00.930559 5493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:00.932243 5493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:00.932298 5493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:00.932370 5493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:00.932403 5493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:00.943235 5493 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:33281
I20260620 06:02:00.943301 5623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:33281 every 8 connection(s)
Jun 20 06:02:00 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:00.944674 5493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:00.949393 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.933383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:38009 (local address 127.3.115.62:34993)
0620 06:02:00.933588 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:00.933592 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:00.934972 (+ 1380us) server_negotiation.cc:407] Connection header received
0620 06:02:00.936381 (+ 1409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:00.936385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:00.936444 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:00.936524 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:00.938373 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.939278 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.940877 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.941285 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.941439 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.945243 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:00.945247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:00.945251 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:00.945284 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:00.947039 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.947660 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.947664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.947666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.947768 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.948136 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.948140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.948142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.948350 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:00.948461 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:00.949091 (+ 630us) server_negotiation.cc:299] Negotiation successful
0620 06:02:00.949225 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 06:02:00.950315 5517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.934189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.3.115.2:38009)
0620 06:02:00.934779 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:00.934843 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:00.936094 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:00.936764 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:00.936774 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:00.937364 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:00.938170 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.938194 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.939460 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.939466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.940728 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.940740 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.941510 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.941514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.941756 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.942585 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:00.942603 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:00.944959 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:00.947196 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.947203 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.947219 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.947534 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.947863 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.947867 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.947869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.948028 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.948459 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:00.948466 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:00.948832 (+ 366us) client_negotiation.cc:769] Sending connection context
0620 06:02:00.949034 (+ 202us) client_negotiation.cc:240] Negotiation successful
0620 06:02:00.949439 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":138,"threads_started":1}
I20260620 06:02:00.951666 5624 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34993
I20260620 06:02:00.951857 5624 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:00.952427 5624 heartbeater.cc:511] Master 127.3.115.62:34993 requested a full tablet report, sending...
I20260620 06:02:00.953266 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 5493
I20260620 06:02:00.953428 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:00.953696 5281 ts_manager.cc:194] Registered new tserver with Master: 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:00.953719 3532 external_mini_cluster.cc:1428] Setting key 5c7131f17cdcaeacc3182c3eef6ee33e
I20260620 06:02:00.954501 5281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:38009
I20260620 06:02:00.963994 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.954479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:48755 (local address 127.3.115.62:34993)
0620 06:02:00.954595 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:00.954602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:00.955039 (+ 437us) server_negotiation.cc:407] Connection header received
0620 06:02:00.955233 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:00.955236 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:00.955286 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:00.955437 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:00.955929 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.956474 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.958029 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.958320 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.958437 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.959791 (+ 1354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:00.959797 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:00.959800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:00.959834 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:00.961993 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.962506 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.962510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.962512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.962603 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:00.962915 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:00.962919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:00.962920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:00.963284 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:00.963400 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:00.963669 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 06:02:00.963809 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:02:00.964035 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:00.954381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.3.115.2:48755)
0620 06:02:00.954933 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:00.954948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:00.955113 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:00.955453 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:00.955456 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:00.955690 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:00.955793 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.955799 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.956605 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.956608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.957834 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:00.957858 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:00.958488 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:00.958491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:00.958572 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:00.959224 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:00.959237 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:00.959611 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:00.962138 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.962142 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.962144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.962377 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.962705 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:00.962709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:00.962711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:00.962807 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:00.963387 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:00.963390 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:00.963498 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 06:02:00.963611 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:02:00.963813 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":4,"thread_start_us":109,"threads_started":1}
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:00.985038 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34993
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:01.100109 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:01.100395 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:01.100445 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:01.104918 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:01.104990 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:01.105050 5633 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 06:02:01.105085 5633 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:02:01.105145 5633 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:01.110615 5633 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:34993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:01.111766 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:01.112943 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:01.122884 5641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:01.122898 5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:01.122943 5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:01.123303 5633 server_base.cc:1034] running on GCE node
I20260620 06:02:01.123776 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:01.124488 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:01.125689 5633 hybrid_clock.cc:648] HybridClock initialized: now 1781935321125678 us; error 30 us; skew 500 ppm
Jun 20 06:02:01 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:01.128621 5633 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:01.129585 5633 webserver.cc:460] Webserver started at http://127.3.115.3:39165/ using document root <none> and password file <none>
I20260620 06:02:01.130162 5633 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:01.130254 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:01.130501 5633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:01.132977 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "3239af37a9584403b4bcffa0756a6a0c"
format_stamp: "Formatted at 2026-06-20 06:02:01 on dist-test-slave-4h14"
server_key: "51bb777db186e9c82009431326cbbe89"
server_key_iv: "2a37e70a5c9f532349c8986f20ab2407"
server_key_version: "encryptionkey@0"
I20260620 06:02:01.133581 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "3239af37a9584403b4bcffa0756a6a0c"
format_stamp: "Formatted at 2026-06-20 06:02:01 on dist-test-slave-4h14"
server_key: "51bb777db186e9c82009431326cbbe89"
server_key_iv: "2a37e70a5c9f532349c8986f20ab2407"
server_key_version: "encryptionkey@0"
I20260620 06:02:01.137562 5633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:01.140049 5648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:01.141549 5633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:01.141700 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "3239af37a9584403b4bcffa0756a6a0c"
format_stamp: "Formatted at 2026-06-20 06:02:01 on dist-test-slave-4h14"
server_key: "51bb777db186e9c82009431326cbbe89"
server_key_iv: "2a37e70a5c9f532349c8986f20ab2407"
server_key_version: "encryptionkey@0"
I20260620 06:02:01.141819 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:01.189185 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:01.192909 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:01.193122 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:01.194623 5633 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:01.194701 5633 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:01.194772 5633 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:01.194816 5633 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:02:01 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935321, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:01.206430 5633 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:34429
I20260620 06:02:01.206476 5763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:34429 every 8 connection(s)
I20260620 06:02:01.207625 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:01.210250 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.195802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:48339 (local address 127.3.115.62:34993)
0620 06:02:01.195939 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.195943 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.197167 (+ 1224us) server_negotiation.cc:407] Connection header received
0620 06:02:01.198505 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.198508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.198572 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.198684 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:01.200358 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.201221 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.202889 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.203239 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.203361 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.206410 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:01.206414 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:01.206417 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:01.206451 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:01.208115 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.208790 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.208794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.208795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.208877 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.209160 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.209162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.209164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.209336 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:01.209433 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.210010 (+ 577us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.210130 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:02:01.211043 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.196421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.3.115.3:48339)
0620 06:02:01.196983 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.197030 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.198256 (+ 1226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.198878 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.198886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.199423 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:01.200206 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.200230 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.201339 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.201343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.202758 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.202768 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.203416 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.203420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.203642 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.204451 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:01.204474 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:01.206204 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:01.208254 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:01.208263 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:01.208281 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:01.208647 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:01.208952 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:01.208954 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:01.208956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:01.209077 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:01.209422 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:01.209426 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:01.209653 (+ 227us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.209851 (+ 198us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.210114 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":158,"threads_started":1}
I20260620 06:02:01.212446 5764 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34993
I20260620 06:02:01.212630 5764 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:01.212767 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 5633
I20260620 06:02:01.212888 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:01.213265 3532 external_mini_cluster.cc:1428] Setting key 7b915d579bacc3e20a2369390ce194a3
I20260620 06:02:01.213459 5764 heartbeater.cc:511] Master 127.3.115.62:34993 requested a full tablet report, sending...
I20260620 06:02:01.214613 5281 ts_manager.cc:194] Registered new tserver with Master: 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429)
I20260620 06:02:01.215464 5281 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:48339
I20260620 06:02:01.215593 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:02:01.223196 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.214741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:35763 (local address 127.3.115.62:34993)
0620 06:02:01.214979 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.214986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.215213 (+ 227us) server_negotiation.cc:407] Connection header received
0620 06:02:01.215368 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.215371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.215414 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.215498 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:01.216011 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.216532 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.217838 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.218139 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.218225 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.219781 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:01.219783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:01.219786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:01.219809 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:01.221400 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.221808 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.221813 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.221815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.221915 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.222169 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.222175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.222180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.222493 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:01.222613 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.222866 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.223007 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260620 06:02:01.223194 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.214691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.3.115.3:35763)
0620 06:02:01.215134 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.215149 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.215281 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.215533 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.215539 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.215756 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:01.215876 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.215883 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.216684 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.216690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.217686 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.217696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.218262 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.218268 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.218389 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.218989 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:01.219005 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:01.219635 (+ 630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:01.221488 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:01.221494 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:01.221499 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:01.221708 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:01.222009 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:01.222012 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:01.222013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:01.222084 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:01.222601 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:01.222604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:01.222686 (+ 82us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.222774 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.222900 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":87,"threads_started":1}
Jun 20 06:02:01 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:02:01 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:01.244832 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.235895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54698 (local address 127.3.115.62:34993)
0620 06:02:01.236050 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.236054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.236068 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:01.236111 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.236113 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.236154 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.236293 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:01.237135 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.237673 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.238543 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.238819 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.238939 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.241589 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:01.241592 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:01.241594 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:01.241622 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:01.243150 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.243580 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.243582 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.243584 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.243664 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.243970 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.243972 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.243973 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.244154 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:01.244259 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.244505 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.244606 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:02:01.248173 5281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:01.251479 5281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:01.266607 5781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.260738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:34429 (local address 127.0.0.1:34144)
0620 06:02:01.261182 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.261237 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.261376 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.262085 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.262091 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.262113 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.262247 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.262253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.263521 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.263526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.264693 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.264704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.266013 (+ 1309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.266016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.266128 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.266168 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.266282 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.266415 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":26,"thread_start_us":99,"threads_started":1}
I20260620 06:02:01.266839 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.261769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.3.115.2:33281)
0620 06:02:01.262095 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.262113 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.262138 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:02:01.262212 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.262217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.262400 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.262821 (+ 421us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.262965 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.264158 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.265493 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.266191 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.266320 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.266500 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.266551 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.266655 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":103,"threads_started":1}
I20260620 06:02:01.267082 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.260857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34144 (local address 127.3.115.3:34429)
0620 06:02:01.261562 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.261575 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.261599 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:02:01.261677 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.261683 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.261919 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.262085 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.262390 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.263392 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.264823 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.265673 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.265768 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.265951 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.266819 (+ 868us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.266935 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":114,"threads_started":1}
I20260620 06:02:01.267169 5784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.261540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35353 (local address 127.0.0.1:36554)
0620 06:02:01.261749 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.261761 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.261835 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.262791 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.262794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.262809 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.262913 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.262919 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.264357 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.264362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.265503 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.265513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.266576 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.266580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.266690 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.266918 (+ 228us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.266964 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.267029 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260620 06:02:01.267215 5782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.260803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33281 (local address 127.0.0.1:55076)
0620 06:02:01.261460 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.261475 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.261563 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.262556 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.262559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.262576 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.262671 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.262676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.264297 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.264300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.265356 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.265365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.266316 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.266320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.266409 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.266430 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.266493 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.266565 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260620 06:02:01.267853 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.262031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36554 (local address 127.3.115.1:35353)
0620 06:02:01.262380 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.262387 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.262407 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:02:01.262481 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.262488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.262655 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.263070 (+ 415us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.263200 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.264268 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.265651 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.266451 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.267452 (+ 1001us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.267589 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.267634 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.267722 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":71,"threads_started":1}
I20260620 06:02:01.270123 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 93e6020a1bba4a14927f7c4a7682cf53 (DEFAULT_TABLE table=test-table [id=a9cc9b8f38804df18712a52634a1d443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:01.270190 5543 tablet_service.cc:1467] Processing CreateTablet for tablet 93e6020a1bba4a14927f7c4a7682cf53 (DEFAULT_TABLE table=test-table [id=a9cc9b8f38804df18712a52634a1d443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:01.271262 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e6020a1bba4a14927f7c4a7682cf53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.271265 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e6020a1bba4a14927f7c4a7682cf53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.271317 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 93e6020a1bba4a14927f7c4a7682cf53 (DEFAULT_TABLE table=test-table [id=a9cc9b8f38804df18712a52634a1d443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:01.272297 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93e6020a1bba4a14927f7c4a7682cf53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.278640 5787 tablet_bootstrap.cc:492] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Bootstrap starting.
I20260620 06:02:01.279721 5788 tablet_bootstrap.cc:492] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Bootstrap starting.
I20260620 06:02:01.279721 5789 tablet_bootstrap.cc:492] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Bootstrap starting.
I20260620 06:02:01.281189 5787 tablet_bootstrap.cc:654] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.282183 5789 tablet_bootstrap.cc:654] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.282225 5787 log.cc:824] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:01.283107 5789 log.cc:824] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:01.283669 5788 tablet_bootstrap.cc:654] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.284965 5788 log.cc:824] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:01.285439 5789 tablet_bootstrap.cc:492] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: No bootstrap required, opened a new log
I20260620 06:02:01.285640 5789 ts_tablet_manager.cc:1397] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260620 06:02:01.285948 5787 tablet_bootstrap.cc:492] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: No bootstrap required, opened a new log
I20260620 06:02:01.286162 5787 ts_tablet_manager.cc:1397] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260620 06:02:01.287866 5788 tablet_bootstrap.cc:492] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: No bootstrap required, opened a new log
I20260620 06:02:01.288043 5788 ts_tablet_manager.cc:1397] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20260620 06:02:01.289986 5789 raft_consensus.cc:348] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.290283 5789 raft_consensus.cc:374] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.290335 5789 raft_consensus.cc:729] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847d2979047844d7a9bbbaee2165e163, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.290830 5789 consensus_queue.cc:260] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.290228 5787 raft_consensus.cc:348] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.291615 5787 raft_consensus.cc:374] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.291685 5787 raft_consensus.cc:729] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.291952 5789 ts_tablet_manager.cc:1428] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260620 06:02:01.292191 5787 consensus_queue.cc:260] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.292352 5624 heartbeater.cc:503] Master 127.3.115.62:34993 was elected leader, sending a full tablet report...
I20260620 06:02:01.292637 5788 raft_consensus.cc:348] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.292938 5788 raft_consensus.cc:374] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.292997 5788 raft_consensus.cc:729] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.293210 5764 heartbeater.cc:503] Master 127.3.115.62:34993 was elected leader, sending a full tablet report...
I20260620 06:02:01.293259 5787 ts_tablet_manager.cc:1428] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 06:02:01.293941 5788 consensus_queue.cc:260] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.294584 5484 heartbeater.cc:503] Master 127.3.115.62:34993 was elected leader, sending a full tablet report...
I20260620 06:02:01.296126 5788 ts_tablet_manager.cc:1428] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Time spent starting tablet: real 0.008s user 0.001s sys 0.006s
I20260620 06:02:01.424502 5795 raft_consensus.cc:482] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:01.424814 5795 raft_consensus.cc:504] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.426021 5795 leader_election.cc:288] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429), 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:01.430274 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.426547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:42157 (local address 127.3.115.2:33281)
0620 06:02:01.426683 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.426687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.426879 (+ 192us) server_negotiation.cc:407] Connection header received
0620 06:02:01.427056 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.427059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.427112 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.427200 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.427463 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.427997 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.429307 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.429697 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.429818 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.429882 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.430070 (+ 188us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.430139 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:02:01.430279 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.426455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:60039 (local address 127.3.115.3:34429)
0620 06:02:01.426599 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.426604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.427058 (+ 454us) server_negotiation.cc:407] Connection header received
0620 06:02:01.427215 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.427218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.427271 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.427356 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.427632 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.428158 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.429316 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.429727 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.429846 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.429912 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.430087 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.430152 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:02:01.430307 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.426319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33281 (local address 127.3.115.1:42157)
0620 06:02:01.426766 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.426779 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.426935 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.427211 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.427214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.427229 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.427354 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.427361 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.428112 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.428117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.429182 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.429191 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.429823 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.429826 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.429930 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.429957 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.430055 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.430133 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":65,"thread_start_us":151,"threads_started":1}
I20260620 06:02:01.430308 5797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.426319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:34429 (local address 127.3.115.1:60039)
0620 06:02:01.426927 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.426941 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.427089 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.427372 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.427375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.427390 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.427513 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.427518 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.428270 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.428276 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.429196 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.429205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.429846 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.429850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.429968 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.429985 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.430072 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.430142 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":159,"thread_start_us":81,"threads_started":1}
I20260620 06:02:01.431095 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6020a1bba4a14927f7c4a7682cf53" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3239af37a9584403b4bcffa0756a6a0c" is_pre_election: true
I20260620 06:02:01.431094 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6020a1bba4a14927f7c4a7682cf53" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163" is_pre_election: true
I20260620 06:02:01.431404 5718 raft_consensus.cc:2454] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 0.
I20260620 06:02:01.431418 5578 raft_consensus.cc:2454] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 0.
I20260620 06:02:01.431972 5369 leader_election.cc:302] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, 847d2979047844d7a9bbbaee2165e163, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:01.432255 5795 raft_consensus.cc:2781] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:01.432350 5795 raft_consensus.cc:482] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:01.432427 5795 raft_consensus.cc:3037] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.433615 5795 raft_consensus.cc:504] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.433979 5795 leader_election.cc:288] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 election: Requested vote from peers 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429), 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:01.434366 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6020a1bba4a14927f7c4a7682cf53" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3239af37a9584403b4bcffa0756a6a0c"
I20260620 06:02:01.434419 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93e6020a1bba4a14927f7c4a7682cf53" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163"
I20260620 06:02:01.434520 5718 raft_consensus.cc:3037] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.434542 5578 raft_consensus.cc:3037] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.435568 5718 raft_consensus.cc:2454] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 1.
I20260620 06:02:01.435603 5578 raft_consensus.cc:2454] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 1.
I20260620 06:02:01.435897 5369 leader_election.cc:302] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:01.436118 5795 raft_consensus.cc:2781] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:01.436379 5795 raft_consensus.cc:686] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 LEADER]: Becoming Leader. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Running, Role: LEADER
I20260620 06:02:01.436786 5795 consensus_queue.cc:237] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } }
I20260620 06:02:01.440294 5279 catalog_manager.cc:5495] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 reported cstate change: term changed from 0 to 1, leader changed from <none> to edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "edfeac70e44e4bf0871fff45cf2eb464" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } health_report { overall_health: HEALTHY } } }
W20260620 06:02:01.446542 5625 tablet.cc:2369] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:02:01.450248 5485 tablet.cc:2369] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:02:01.458964 5765 tablet.cc:2369] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 06:02:01 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935320, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:01.483201 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.474450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34993 (local address 127.0.0.1:55578)
0620 06:02:01.474769 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.474782 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.474897 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.475136 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.475138 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.475525 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:01.475615 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.475620 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.476333 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.476335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.476987 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.476997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.477472 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.477475 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.477534 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.478147 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:01.478160 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:01.479638 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:01.481493 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:01.481506 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:01.481513 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:01.481756 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:01.482048 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:01.482051 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:01.482053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:01.482141 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:01.482548 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:01.482558 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:01.482687 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.482812 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.483001 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":139,"threads_started":1}
I20260620 06:02:01.483201 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.474501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55578 (local address 127.3.115.62:34993)
0620 06:02:01.474649 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.474654 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.474875 (+ 221us) server_negotiation.cc:407] Connection header received
0620 06:02:01.474985 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.474987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.475029 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.475102 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:01.475720 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.476232 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.477139 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.477372 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.477455 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.479790 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:01.479794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:01.479796 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:01.479821 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:01.481365 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.481862 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.481867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.481869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.481948 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:01.482235 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:01.482237 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:01.482239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:01.482429 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:01.482535 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.482856 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.483059 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":95,"server-negotiator.queue_time_us":58}
I20260620 06:02:01.487673 5279 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55578:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:02:01.488422 5279 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:01.493613 5543 tablet_service.cc:1467] Processing CreateTablet for tablet ac817f1e37b54c18934e74b0ee7ebdec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e98ec0d9c4964f3fafba440e52e57f39]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:02:01.493572 5418 tablet_service.cc:1467] Processing CreateTablet for tablet ac817f1e37b54c18934e74b0ee7ebdec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e98ec0d9c4964f3fafba440e52e57f39]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:02:01.493732 5698 tablet_service.cc:1467] Processing CreateTablet for tablet ac817f1e37b54c18934e74b0ee7ebdec (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e98ec0d9c4964f3fafba440e52e57f39]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:02:01.494392 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac817f1e37b54c18934e74b0ee7ebdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.494392 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac817f1e37b54c18934e74b0ee7ebdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.494393 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac817f1e37b54c18934e74b0ee7ebdec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.497253 5789 tablet_bootstrap.cc:492] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: Bootstrap starting.
I20260620 06:02:01.497298 5788 tablet_bootstrap.cc:492] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: Bootstrap starting.
I20260620 06:02:01.497298 5787 tablet_bootstrap.cc:492] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: Bootstrap starting.
I20260620 06:02:01.498076 5789 tablet_bootstrap.cc:654] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.498359 5787 tablet_bootstrap.cc:654] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.498711 5788 tablet_bootstrap.cc:654] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.500022 5789 tablet_bootstrap.cc:492] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: No bootstrap required, opened a new log
I20260620 06:02:01.500084 5788 tablet_bootstrap.cc:492] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: No bootstrap required, opened a new log
I20260620 06:02:01.500140 5789 ts_tablet_manager.cc:1397] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260620 06:02:01.500174 5788 ts_tablet_manager.cc:1397] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 06:02:01.500300 5787 tablet_bootstrap.cc:492] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: No bootstrap required, opened a new log
I20260620 06:02:01.500385 5787 ts_tablet_manager.cc:1397] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:01.500737 5788 raft_consensus.cc:348] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.500869 5788 raft_consensus.cc:374] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.500916 5788 raft_consensus.cc:729] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.500941 5787 raft_consensus.cc:348] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.501067 5787 raft_consensus.cc:374] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.501049 5788 consensus_queue.cc:260] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.501108 5787 raft_consensus.cc:729] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.501088 5789 raft_consensus.cc:348] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.501219 5789 raft_consensus.cc:374] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.501262 5789 raft_consensus.cc:729] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847d2979047844d7a9bbbaee2165e163, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.501232 5787 consensus_queue.cc:260] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.501338 5788 ts_tablet_manager.cc:1428] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:01.501395 5789 consensus_queue.cc:260] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.501511 5787 ts_tablet_manager.cc:1428] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:01.501688 5793 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.501842 5793 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.501788 5795 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.501917 5795 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.502122 5794 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.502259 5794 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.501668 5789 ts_tablet_manager.cc:1428] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260620 06:02:01.561470 5794 raft_consensus.cc:482] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:01.561605 5794 raft_consensus.cc:504] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.562672 5794 leader_election.cc:288] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353), 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:01.567341 5657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.562966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33281 (local address 127.3.115.3:60447)
0620 06:02:01.563076 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.563091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.563187 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.563469 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.563474 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.563497 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.563635 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.563643 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.564630 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.564633 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.565916 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.565926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.566820 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.566822 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.566949 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.567011 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.567090 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.567172 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":23}
I20260620 06:02:01.567358 5809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.562981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35353 (local address 127.3.115.3:38277)
0620 06:02:01.563468 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.563481 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.563572 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.563931 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.563932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.563947 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:01.564038 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.564042 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.564954 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.564957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.566075 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.566082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.566967 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.566970 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.567089 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.567109 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.567173 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.567223 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":124,"thread_start_us":134,"threads_started":1}
I20260620 06:02:01.567410 5785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.563043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:60447 (local address 127.3.115.2:33281)
0620 06:02:01.563203 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.563208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.563225 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:02:01.563317 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.563324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.563367 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.563428 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.563766 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.564520 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.566047 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.566700 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.566808 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.566881 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.567184 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.567273 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:02:01.567410 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.563136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38277 (local address 127.3.115.1:35353)
0620 06:02:01.563274 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.563279 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.563574 (+ 295us) server_negotiation.cc:407] Connection header received
0620 06:02:01.563668 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.563670 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.563745 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.563884 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:01.564149 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.564839 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.566176 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.566830 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.566935 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.566977 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.567224 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.567297 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:02:01.567842 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac817f1e37b54c18934e74b0ee7ebdec" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163" is_pre_election: true
I20260620 06:02:01.568024 5578 raft_consensus.cc:2454] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 0.
I20260620 06:02:01.567997 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac817f1e37b54c18934e74b0ee7ebdec" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfeac70e44e4bf0871fff45cf2eb464" is_pre_election: true
I20260620 06:02:01.568212 5438 raft_consensus.cc:2454] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 0.
I20260620 06:02:01.568435 5652 leader_election.cc:302] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, 847d2979047844d7a9bbbaee2165e163; no voters:
I20260620 06:02:01.568786 5794 raft_consensus.cc:2781] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:01.568850 5794 raft_consensus.cc:482] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:01.568897 5794 raft_consensus.cc:3037] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.569896 5794 raft_consensus.cc:504] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.570338 5794 leader_election.cc:288] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 election: Requested vote from peers edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353), 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:01.570652 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac817f1e37b54c18934e74b0ee7ebdec" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfeac70e44e4bf0871fff45cf2eb464"
I20260620 06:02:01.570776 5438 raft_consensus.cc:3037] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.570739 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac817f1e37b54c18934e74b0ee7ebdec" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163"
I20260620 06:02:01.570887 5578 raft_consensus.cc:3037] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.571861 5438 raft_consensus.cc:2454] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 1.
I20260620 06:02:01.572007 5578 raft_consensus.cc:2454] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 1.
I20260620 06:02:01.572160 5649 leader_election.cc:302] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:01.572335 5794 raft_consensus.cc:2781] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:01.572602 5794 raft_consensus.cc:686] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [term 1 LEADER]: Becoming Leader. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Running, Role: LEADER
I20260620 06:02:01.572990 5794 consensus_queue.cc:237] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.575148 5810 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: TxnStatusTablet state changed. Reason: New leader 3239af37a9584403b4bcffa0756a6a0c. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.575286 5810 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:02:01.575608 5812 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:02:01.575871 5812 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:02:01.576403 5281 catalog_manager.cc:5495] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3). New cstate: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } health_report { overall_health: HEALTHY } } }
I20260620 06:02:01.602304 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.598693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55648 (local address 127.3.115.3:34429)
0620 06:02:01.598828 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.598832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.598843 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:02:01.598913 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.598915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.598964 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.599075 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:01.599324 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.599828 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.600780 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.601001 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.601113 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.601403 (+ 290us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.601893 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.602020 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.602086 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.602136 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:02:01.602351 5805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.597962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:34429 (local address 127.0.0.1:55648)
0620 06:02:01.598195 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.598215 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.598347 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.599091 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.599094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.599102 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:01.599213 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.599218 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.599968 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.599972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.600599 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.600608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.601114 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.601118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.601213 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.601257 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.601991 (+ 734us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.602007 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.602077 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.602187 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 06:02:01.607730 5578 raft_consensus.cc:1261] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Refusing update from remote peer 3239af37a9584403b4bcffa0756a6a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:01.607733 5438 raft_consensus.cc:1261] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Refusing update from remote peer 3239af37a9584403b4bcffa0756a6a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:01.608456 5810 consensus_queue.cc:1035] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:01.608647 5794 consensus_queue.cc:1035] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:01.612507 5795 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: TxnStatusTablet state changed. Reason: New leader 3239af37a9584403b4bcffa0756a6a0c. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.612639 5795 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.613502 5793 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: TxnStatusTablet state changed. Reason: New leader 3239af37a9584403b4bcffa0756a6a0c. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.613623 5793 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.616427 5794 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.616559 5794 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:02:01.616770 5795 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.616897 5795 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P edfeac70e44e4bf0871fff45cf2eb464: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.616905 5794 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.616990 5794 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 3239af37a9584403b4bcffa0756a6a0c: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:02:01.617383 5815 mvcc.cc:205] Tried to move back new op lower bound from 7298807077298544640 to 7298807077165187072. Current Snapshot: MvccSnapshot[applied={T|T < 7298807077298544640}]
I20260620 06:02:01.617420 5793 tablet_replica.cc:439] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } } }
I20260620 06:02:01.617525 5793 tablet_replica.cc:442] T ac817f1e37b54c18934e74b0ee7ebdec P 847d2979047844d7a9bbbaee2165e163: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:01.618062 5817 mvcc.cc:205] Tried to move back new op lower bound from 7298807077298544640 to 7298807077165187072. Current Snapshot: MvccSnapshot[applied={T|T < 7298807077298544640}]
I20260620 06:02:01.619102 5816 mvcc.cc:205] Tried to move back new op lower bound from 7298807077298544640 to 7298807077165187072. Current Snapshot: MvccSnapshot[applied={T|T < 7298807077298544640}]
I20260620 06:02:01.629911 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.625774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34866 (local address 127.3.115.1:35353)
0620 06:02:01.625895 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.625900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.625928 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:02:01.626100 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.626104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.626168 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.626282 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:01.626495 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.627291 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.628202 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.628573 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.628710 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.628973 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.629546 (+ 573us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.629664 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.629705 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.629765 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:02:01.641700 5826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.638011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:34429 (local address 127.3.115.1:50523)
0620 06:02:01.638426 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.638439 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.638557 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.638813 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.638816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.638823 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:01.638969 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.638974 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.639717 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.639723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.640363 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.640372 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.640904 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.640908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.641016 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.641061 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.641393 (+ 332us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.641400 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.641466 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.641545 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":135,"threads_started":1}
I20260620 06:02:01.641669 5783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.638120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:50523 (local address 127.3.115.3:34429)
0620 06:02:01.638265 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.638269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.638506 (+ 237us) server_negotiation.cc:407] Connection header received
0620 06:02:01.638671 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.638673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.638718 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.638803 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:01.639075 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.639603 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.640492 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.640784 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.640896 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.641222 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.641306 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.641396 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.641475 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.641522 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:02:01.648841 5578 raft_consensus.cc:1261] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Refusing update from remote peer edfeac70e44e4bf0871fff45cf2eb464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:01.649206 5718 raft_consensus.cc:1261] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Refusing update from remote peer edfeac70e44e4bf0871fff45cf2eb464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:01.649350 5798 consensus_queue.cc:1035] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:01.649797 5798 consensus_queue.cc:1035] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:01.673012 5770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.668851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35353 (local address 127.3.115.3:38665)
0620 06:02:01.669019 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:01.669043 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:01.669127 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:01.669376 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:01.669378 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:01.669383 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:01.669481 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.669486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.670565 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.670568 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.671494 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:01.671505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.672218 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.672223 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:01.672303 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.672361 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.672741 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.672747 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:02:01.672818 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:02:01.672893 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 06:02:01.673048 5786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:01.668915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38665 (local address 127.3.115.1:35353)
0620 06:02:01.669048 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:01.669053 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:01.669087 (+ 34us) server_negotiation.cc:407] Connection header received
0620 06:02:01.669200 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:01.669203 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:01.669258 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:01.669337 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:01.669590 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.670431 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.671620 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:01.671986 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:01.672096 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:01.672522 (+ 426us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:01.672620 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:01.672735 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:01.672843 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 06:02:01.672909 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:02:01.702600 5281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:01.702805 5281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:01.706140 5281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Sending DeleteTablet for 3 replicas of tablet 93e6020a1bba4a14927f7c4a7682cf53
I20260620 06:02:01.707268 5543 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e6020a1bba4a14927f7c4a7682cf53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:01 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55076
I20260620 06:02:01.708374 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e6020a1bba4a14927f7c4a7682cf53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:01 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:36554
I20260620 06:02:01.709124 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 93e6020a1bba4a14927f7c4a7682cf53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:01 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:34144
I20260620 06:02:01.710345 5281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:01.710861 5281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:01.712780 5834 tablet_replica.cc:330] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: stopping tablet replica
I20260620 06:02:01.713135 5834 raft_consensus.cc:2229] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:01.713366 5834 raft_consensus.cc:2258] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:01.716626 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 84d102b09a16496a8a1dd46caeb05230 (DEFAULT_TABLE table=test-table [id=e0a590c6be3b439a90c02720474f8844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:01.716928 5835 tablet_replica.cc:330] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: stopping tablet replica
I20260620 06:02:01.716971 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d102b09a16496a8a1dd46caeb05230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.717298 5835 raft_consensus.cc:2229] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:01.717687 5835 raft_consensus.cc:2258] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:01.718003 5836 tablet_replica.cc:330] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: stopping tablet replica
I20260620 06:02:01.718288 5836 raft_consensus.cc:2229] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:01.718453 5836 raft_consensus.cc:2258] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:01.719715 5788 tablet_bootstrap.cc:492] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Bootstrap starting.
I20260620 06:02:01.719753 5835 ts_tablet_manager.cc:1905] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:01.720773 5788 tablet_bootstrap.cc:654] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.721843 5543 tablet_service.cc:1467] Processing CreateTablet for tablet 84d102b09a16496a8a1dd46caeb05230 (DEFAULT_TABLE table=test-table [id=e0a590c6be3b439a90c02720474f8844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:01.722141 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d102b09a16496a8a1dd46caeb05230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.722573 5835 ts_tablet_manager.cc:1918] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:01.722671 5835 log.cc:1196] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/93e6020a1bba4a14927f7c4a7682cf53
I20260620 06:02:01.722986 5835 ts_tablet_manager.cc:1939] T 93e6020a1bba4a14927f7c4a7682cf53 P edfeac70e44e4bf0871fff45cf2eb464: Deleting consensus metadata
I20260620 06:02:01.724231 5265 catalog_manager.cc:4841] TS edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353): tablet 93e6020a1bba4a14927f7c4a7682cf53 (table test-table [id=a9cc9b8f38804df18712a52634a1d443]) successfully deleted
I20260620 06:02:01.724607 5789 tablet_bootstrap.cc:492] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Bootstrap starting.
I20260620 06:02:01.725600 5789 tablet_bootstrap.cc:654] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.729200 5836 ts_tablet_manager.cc:1905] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:01.731091 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 84d102b09a16496a8a1dd46caeb05230 (DEFAULT_TABLE table=test-table [id=e0a590c6be3b439a90c02720474f8844]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:01.731398 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84d102b09a16496a8a1dd46caeb05230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:01.734055 5787 tablet_bootstrap.cc:492] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Bootstrap starting.
I20260620 06:02:01.734562 5834 ts_tablet_manager.cc:1905] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:01.735140 5787 tablet_bootstrap.cc:654] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:01.736256 5836 ts_tablet_manager.cc:1918] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:01.736364 5836 log.cc:1196] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/93e6020a1bba4a14927f7c4a7682cf53
I20260620 06:02:01.736721 5836 ts_tablet_manager.cc:1939] T 93e6020a1bba4a14927f7c4a7682cf53 P 3239af37a9584403b4bcffa0756a6a0c: Deleting consensus metadata
I20260620 06:02:01.737954 5265 catalog_manager.cc:4841] TS 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429): tablet 93e6020a1bba4a14927f7c4a7682cf53 (table test-table [id=a9cc9b8f38804df18712a52634a1d443]) successfully deleted
I20260620 06:02:01.738857 5787 tablet_bootstrap.cc:492] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: No bootstrap required, opened a new log
I20260620 06:02:01.738952 5787 ts_tablet_manager.cc:1397] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 06:02:01.739452 5834 ts_tablet_manager.cc:1918] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:01.739430 5787 raft_consensus.cc:348] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.739543 5787 raft_consensus.cc:374] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.739584 5787 raft_consensus.cc:729] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.739707 5787 consensus_queue.cc:260] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.739977 5787 ts_tablet_manager.cc:1428] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:01.739575 5834 log.cc:1196] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/93e6020a1bba4a14927f7c4a7682cf53
I20260620 06:02:01.742308 5834 ts_tablet_manager.cc:1939] T 93e6020a1bba4a14927f7c4a7682cf53 P 847d2979047844d7a9bbbaee2165e163: Deleting consensus metadata
I20260620 06:02:01.744850 5268 catalog_manager.cc:4841] TS 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281): tablet 93e6020a1bba4a14927f7c4a7682cf53 (table test-table [id=a9cc9b8f38804df18712a52634a1d443]) successfully deleted
I20260620 06:02:01.746169 5788 tablet_bootstrap.cc:492] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: No bootstrap required, opened a new log
I20260620 06:02:01.746282 5788 ts_tablet_manager.cc:1397] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.001s
I20260620 06:02:01.746838 5788 raft_consensus.cc:348] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.746996 5788 raft_consensus.cc:374] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.747027 5788 raft_consensus.cc:729] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.747154 5788 consensus_queue.cc:260] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.747409 5788 ts_tablet_manager.cc:1428] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:01.748880 5789 tablet_bootstrap.cc:492] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: No bootstrap required, opened a new log
I20260620 06:02:01.748989 5789 ts_tablet_manager.cc:1397] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.001s
I20260620 06:02:01.749436 5789 raft_consensus.cc:348] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.749559 5789 raft_consensus.cc:374] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:01.749588 5789 raft_consensus.cc:729] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847d2979047844d7a9bbbaee2165e163, State: Initialized, Role: FOLLOWER
I20260620 06:02:01.749698 5789 consensus_queue.cc:260] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.749948 5789 ts_tablet_manager.cc:1428] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:01.767066 5829 raft_consensus.cc:482] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:01.767215 5829 raft_consensus.cc:504] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.767688 5829 leader_election.cc:288] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281), 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429)
I20260620 06:02:01.768044 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d102b09a16496a8a1dd46caeb05230" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163" is_pre_election: true
I20260620 06:02:01.768139 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d102b09a16496a8a1dd46caeb05230" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3239af37a9584403b4bcffa0756a6a0c" is_pre_election: true
I20260620 06:02:01.768193 5578 raft_consensus.cc:2454] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 0.
I20260620 06:02:01.768294 5717 raft_consensus.cc:2454] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 0.
I20260620 06:02:01.768524 5372 leader_election.cc:302] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 847d2979047844d7a9bbbaee2165e163, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:01.768754 5829 raft_consensus.cc:2781] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:01.768821 5829 raft_consensus.cc:482] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:01.768886 5829 raft_consensus.cc:3037] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.770241 5829 raft_consensus.cc:504] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.770618 5829 leader_election.cc:288] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 election: Requested vote from peers 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281), 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429)
I20260620 06:02:01.771004 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d102b09a16496a8a1dd46caeb05230" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163"
I20260620 06:02:01.771065 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84d102b09a16496a8a1dd46caeb05230" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3239af37a9584403b4bcffa0756a6a0c"
I20260620 06:02:01.771164 5578 raft_consensus.cc:3037] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.771184 5717 raft_consensus.cc:3037] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:01.772221 5578 raft_consensus.cc:2454] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 1.
I20260620 06:02:01.772301 5717 raft_consensus.cc:2454] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 1.
I20260620 06:02:01.772512 5372 leader_election.cc:302] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 847d2979047844d7a9bbbaee2165e163, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:01.772796 5829 raft_consensus.cc:2781] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:01.772889 5829 raft_consensus.cc:686] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 LEADER]: Becoming Leader. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Running, Role: LEADER
I20260620 06:02:01.773025 5829 consensus_queue.cc:237] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:01.774647 5281 catalog_manager.cc:5495] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 reported cstate change: term changed from 0 to 1, leader changed from <none> to edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "edfeac70e44e4bf0871fff45cf2eb464" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } health_report { overall_health: UNKNOWN } } }
I20260620 06:02:01.797453 5577 raft_consensus.cc:1261] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Refusing update from remote peer edfeac70e44e4bf0871fff45cf2eb464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:01.797446 5717 raft_consensus.cc:1261] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Refusing update from remote peer edfeac70e44e4bf0871fff45cf2eb464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:01.797852 5829 consensus_queue.cc:1035] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:01.798039 5795 consensus_queue.cc:1035] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:02:01.813258 5817 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:02:03.367107 5280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:03.367282 5280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54698:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:03.369668 5280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Sending DeleteTablet for 3 replicas of tablet 84d102b09a16496a8a1dd46caeb05230
I20260620 06:02:03.370293 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d102b09a16496a8a1dd46caeb05230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:03 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:36554
I20260620 06:02:03.370273 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d102b09a16496a8a1dd46caeb05230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:03 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:34144
I20260620 06:02:03.370308 5543 tablet_service.cc:1514] Processing DeleteTablet for tablet 84d102b09a16496a8a1dd46caeb05230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:03 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55076
I20260620 06:02:03.370635 5870 tablet_replica.cc:330] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: stopping tablet replica
I20260620 06:02:03.370762 5870 raft_consensus.cc:2229] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:03.370886 5870 raft_consensus.cc:2258] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:03.371130 5869 tablet_replica.cc:330] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: stopping tablet replica
I20260620 06:02:03.371258 5869 raft_consensus.cc:2229] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:03.371376 5869 raft_consensus.cc:2258] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:03.371907 5870 ts_tablet_manager.cc:1905] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:03.371979 5869 ts_tablet_manager.cc:1905] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:03.373665 5868 tablet_replica.cc:330] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: stopping tablet replica
I20260620 06:02:03.373816 5868 raft_consensus.cc:2229] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:03.374038 5868 raft_consensus.cc:2258] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:03.374617 5869 ts_tablet_manager.cc:1918] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:02:03.374617 5870 ts_tablet_manager.cc:1918] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:02:03.374701 5869 log.cc:1196] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/84d102b09a16496a8a1dd46caeb05230
I20260620 06:02:03.374701 5870 log.cc:1196] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/84d102b09a16496a8a1dd46caeb05230
I20260620 06:02:03.374984 5869 ts_tablet_manager.cc:1939] T 84d102b09a16496a8a1dd46caeb05230 P 847d2979047844d7a9bbbaee2165e163: Deleting consensus metadata
I20260620 06:02:03.374986 5870 ts_tablet_manager.cc:1939] T 84d102b09a16496a8a1dd46caeb05230 P 3239af37a9584403b4bcffa0756a6a0c: Deleting consensus metadata
I20260620 06:02:03.375787 5268 catalog_manager.cc:4841] TS 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281): tablet 84d102b09a16496a8a1dd46caeb05230 (table test-table [id=e0a590c6be3b439a90c02720474f8844]) successfully deleted
I20260620 06:02:03.375808 5265 catalog_manager.cc:4841] TS 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429): tablet 84d102b09a16496a8a1dd46caeb05230 (table test-table [id=e0a590c6be3b439a90c02720474f8844]) successfully deleted
I20260620 06:02:03.376341 5868 ts_tablet_manager.cc:1905] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:03.378904 5868 ts_tablet_manager.cc:1918] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:02:03.378979 5868 log.cc:1196] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/84d102b09a16496a8a1dd46caeb05230
I20260620 06:02:03.379285 5868 ts_tablet_manager.cc:1939] T 84d102b09a16496a8a1dd46caeb05230 P edfeac70e44e4bf0871fff45cf2eb464: Deleting consensus metadata
I20260620 06:02:03.380007 5265 catalog_manager.cc:4841] TS edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353): tablet 84d102b09a16496a8a1dd46caeb05230 (table test-table [id=e0a590c6be3b439a90c02720474f8844]) successfully deleted
Jun 20 06:02:03 dist-test-slave-4h14 krb5kdc[5236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:02:03 dist-test-slave-4h14 krb5kdc[5236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935323, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:03.404939 5878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:03.390928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55584 (local address 127.3.115.62:34993)
0620 06:02:03.391258 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:03.391265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:03.391279 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:03.391424 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:03.391426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:03.391470 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:03.391552 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:03.392246 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.392880 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.393795 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.394023 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.394105 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:03.400764 (+ 6659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:03.400769 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:03.400771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:03.400807 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:03.402736 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:03.403274 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:03.403277 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:03.403278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:03.403391 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:03.403792 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:03.403796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:03.403797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:03.404029 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:03.404159 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:03.404474 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 06:02:03.404630 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":145,"threads_started":1}
I20260620 06:02:03.407752 5280 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:03.408252 5280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:03.413264 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 9a773e0e397f462196aa7381cbd22e11 (DEFAULT_TABLE table=test-table [id=458a5c6922c24ff2b4f8e5e2e87b1045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:03.413472 5543 tablet_service.cc:1467] Processing CreateTablet for tablet 9a773e0e397f462196aa7381cbd22e11 (DEFAULT_TABLE table=test-table [id=458a5c6922c24ff2b4f8e5e2e87b1045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:03.413611 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a773e0e397f462196aa7381cbd22e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:03.413801 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a773e0e397f462196aa7381cbd22e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:03.413759 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 9a773e0e397f462196aa7381cbd22e11 (DEFAULT_TABLE table=test-table [id=458a5c6922c24ff2b4f8e5e2e87b1045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:03.414032 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a773e0e397f462196aa7381cbd22e11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:03.415740 5879 tablet_bootstrap.cc:492] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Bootstrap starting.
I20260620 06:02:03.416745 5879 tablet_bootstrap.cc:654] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:03.417156 5880 tablet_bootstrap.cc:492] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Bootstrap starting.
I20260620 06:02:03.418056 5880 tablet_bootstrap.cc:654] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:03.418781 5881 tablet_bootstrap.cc:492] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Bootstrap starting.
I20260620 06:02:03.419708 5880 tablet_bootstrap.cc:492] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: No bootstrap required, opened a new log
I20260620 06:02:03.419813 5880 ts_tablet_manager.cc:1397] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:03.419854 5881 tablet_bootstrap.cc:654] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:03.420423 5880 raft_consensus.cc:348] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.420583 5880 raft_consensus.cc:374] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:03.420629 5880 raft_consensus.cc:729] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847d2979047844d7a9bbbaee2165e163, State: Initialized, Role: FOLLOWER
I20260620 06:02:03.420811 5880 consensus_queue.cc:260] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.421237 5881 tablet_bootstrap.cc:492] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: No bootstrap required, opened a new log
I20260620 06:02:03.421272 5880 ts_tablet_manager.cc:1428] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260620 06:02:03.421430 5881 ts_tablet_manager.cc:1397] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:03.421851 5881 raft_consensus.cc:348] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.421976 5881 raft_consensus.cc:374] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:03.422020 5881 raft_consensus.cc:729] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Initialized, Role: FOLLOWER
I20260620 06:02:03.422173 5881 consensus_queue.cc:260] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.422574 5881 ts_tablet_manager.cc:1428] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:03.425319 5879 tablet_bootstrap.cc:492] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: No bootstrap required, opened a new log
I20260620 06:02:03.425411 5879 ts_tablet_manager.cc:1397] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.001s
I20260620 06:02:03.425783 5879 raft_consensus.cc:348] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.425881 5879 raft_consensus.cc:374] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:03.425904 5879 raft_consensus.cc:729] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Initialized, Role: FOLLOWER
I20260620 06:02:03.426065 5879 consensus_queue.cc:260] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.426295 5879 ts_tablet_manager.cc:1428] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:02:03.475607 5863 raft_consensus.cc:482] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:03.475798 5863 raft_consensus.cc:504] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.476266 5863 leader_election.cc:288] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281), 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429)
I20260620 06:02:03.476742 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a773e0e397f462196aa7381cbd22e11" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3239af37a9584403b4bcffa0756a6a0c" is_pre_election: true
I20260620 06:02:03.476787 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a773e0e397f462196aa7381cbd22e11" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163" is_pre_election: true
I20260620 06:02:03.476897 5717 raft_consensus.cc:2454] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 0.
I20260620 06:02:03.477001 5578 raft_consensus.cc:2454] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 0.
I20260620 06:02:03.477241 5369 leader_election.cc:302] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:03.477435 5863 raft_consensus.cc:2781] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:03.477502 5863 raft_consensus.cc:482] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:03.477530 5863 raft_consensus.cc:3037] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:03.478634 5863 raft_consensus.cc:504] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.478986 5863 leader_election.cc:288] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 election: Requested vote from peers 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281), 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429)
I20260620 06:02:03.479460 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a773e0e397f462196aa7381cbd22e11" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163"
I20260620 06:02:03.479468 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a773e0e397f462196aa7381cbd22e11" candidate_uuid: "edfeac70e44e4bf0871fff45cf2eb464" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3239af37a9584403b4bcffa0756a6a0c"
I20260620 06:02:03.479578 5717 raft_consensus.cc:3037] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:03.479586 5578 raft_consensus.cc:3037] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:03.480489 5578 raft_consensus.cc:2454] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 1.
I20260620 06:02:03.480538 5717 raft_consensus.cc:2454] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edfeac70e44e4bf0871fff45cf2eb464 in term 1.
I20260620 06:02:03.480845 5372 leader_election.cc:302] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 847d2979047844d7a9bbbaee2165e163, edfeac70e44e4bf0871fff45cf2eb464; no voters:
I20260620 06:02:03.481027 5863 raft_consensus.cc:2781] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:03.481148 5863 raft_consensus.cc:686] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 LEADER]: Becoming Leader. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Running, Role: LEADER
I20260620 06:02:03.481289 5863 consensus_queue.cc:237] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.482983 5281 catalog_manager.cc:5495] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 reported cstate change: term changed from 0 to 1, leader changed from <none> to edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "edfeac70e44e4bf0871fff45cf2eb464" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } health_report { overall_health: UNKNOWN } } }
I20260620 06:02:03.510672 5888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:03.507238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34878 (local address 127.3.115.1:35353)
0620 06:02:03.507506 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:03.507513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:03.507525 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:02:03.507563 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:03.507565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:03.507603 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:03.507687 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:03.507984 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.508502 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.509433 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.509674 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.509756 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:03.510212 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:03.510297 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:03.510365 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:03.510497 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 06:02:03.510549 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":104,"threads_started":1}
I20260620 06:02:03.517170 5717 raft_consensus.cc:1261] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Refusing update from remote peer edfeac70e44e4bf0871fff45cf2eb464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:03.517170 5578 raft_consensus.cc:1261] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Refusing update from remote peer edfeac70e44e4bf0871fff45cf2eb464: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:03.517653 5863 consensus_queue.cc:1035] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:03.517894 5858 consensus_queue.cc:1035] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [LEADER]: Connected to new peer: Peer: permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:03.549847 5281 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:03.550000 5281 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:03.552141 5281 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Sending DeleteTablet for 3 replicas of tablet 9a773e0e397f462196aa7381cbd22e11
I20260620 06:02:03.552716 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a773e0e397f462196aa7381cbd22e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:03 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:36554
I20260620 06:02:03.552968 5868 tablet_replica.cc:330] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: stopping tablet replica
I20260620 06:02:03.552935 5543 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a773e0e397f462196aa7381cbd22e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:03 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55076
I20260620 06:02:03.553119 5868 raft_consensus.cc:2229] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:03.553157 5869 tablet_replica.cc:330] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: stopping tablet replica
I20260620 06:02:03.553292 5869 raft_consensus.cc:2229] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:03.553333 5868 raft_consensus.cc:2258] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:03.553432 5869 raft_consensus.cc:2258] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:03.553741 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a773e0e397f462196aa7381cbd22e11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:03 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:34144
I20260620 06:02:03.553686 5281 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:02:03.553941 5870 tablet_replica.cc:330] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: stopping tablet replica
I20260620 06:02:03.554028 5870 raft_consensus.cc:2229] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 06:02:03.554113 5281 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:03.554139 5870 raft_consensus.cc:2258] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:03.554207 5868 ts_tablet_manager.cc:1905] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:03.554771 5870 ts_tablet_manager.cc:1905] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:03.556859 5868 ts_tablet_manager.cc:1918] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:03.556933 5868 log.cc:1196] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/9a773e0e397f462196aa7381cbd22e11
I20260620 06:02:03.557160 5870 ts_tablet_manager.cc:1918] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:03.557205 5868 ts_tablet_manager.cc:1939] T 9a773e0e397f462196aa7381cbd22e11 P edfeac70e44e4bf0871fff45cf2eb464: Deleting consensus metadata
I20260620 06:02:03.557199 5869 ts_tablet_manager.cc:1905] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:03.557229 5870 log.cc:1196] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/9a773e0e397f462196aa7381cbd22e11
I20260620 06:02:03.557479 5870 ts_tablet_manager.cc:1939] T 9a773e0e397f462196aa7381cbd22e11 P 3239af37a9584403b4bcffa0756a6a0c: Deleting consensus metadata
I20260620 06:02:03.559298 5543 tablet_service.cc:1467] Processing CreateTablet for tablet ce3fd6fd469442288f159d9ecf8f413c (DEFAULT_TABLE table=test-table [id=bfb5edd29bdd4bbaac27c113ee69434a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:03.559557 5869 ts_tablet_manager.cc:1918] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:03.559561 5543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3fd6fd469442288f159d9ecf8f413c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:03.559666 5869 log.cc:1196] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/9a773e0e397f462196aa7381cbd22e11
I20260620 06:02:03.559899 5869 ts_tablet_manager.cc:1939] T 9a773e0e397f462196aa7381cbd22e11 P 847d2979047844d7a9bbbaee2165e163: Deleting consensus metadata
I20260620 06:02:03.560491 5268 catalog_manager.cc:4841] TS 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281): tablet 9a773e0e397f462196aa7381cbd22e11 (table test-table [id=458a5c6922c24ff2b4f8e5e2e87b1045]) successfully deleted
I20260620 06:02:03.560683 5265 catalog_manager.cc:4841] TS 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429): tablet 9a773e0e397f462196aa7381cbd22e11 (table test-table [id=458a5c6922c24ff2b4f8e5e2e87b1045]) successfully deleted
I20260620 06:02:03.560840 5265 catalog_manager.cc:4841] TS edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353): tablet 9a773e0e397f462196aa7381cbd22e11 (table test-table [id=458a5c6922c24ff2b4f8e5e2e87b1045]) successfully deleted
I20260620 06:02:03.560984 5418 tablet_service.cc:1467] Processing CreateTablet for tablet ce3fd6fd469442288f159d9ecf8f413c (DEFAULT_TABLE table=test-table [id=bfb5edd29bdd4bbaac27c113ee69434a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:03.561223 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3fd6fd469442288f159d9ecf8f413c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:03.561830 5880 tablet_bootstrap.cc:492] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Bootstrap starting.
I20260620 06:02:03.562608 5880 tablet_bootstrap.cc:654] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:03.563318 5879 tablet_bootstrap.cc:492] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464: Bootstrap starting.
I20260620 06:02:03.563803 5698 tablet_service.cc:1467] Processing CreateTablet for tablet ce3fd6fd469442288f159d9ecf8f413c (DEFAULT_TABLE table=test-table [id=bfb5edd29bdd4bbaac27c113ee69434a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:03.564100 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce3fd6fd469442288f159d9ecf8f413c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:03.564368 5879 tablet_bootstrap.cc:654] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:03.565213 5880 tablet_bootstrap.cc:492] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: No bootstrap required, opened a new log
I20260620 06:02:03.565325 5880 ts_tablet_manager.cc:1397] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 06:02:03.565528 5879 tablet_bootstrap.cc:492] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464: No bootstrap required, opened a new log
I20260620 06:02:03.565635 5879 ts_tablet_manager.cc:1397] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:03.565819 5880 raft_consensus.cc:348] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.565946 5880 raft_consensus.cc:374] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:03.565979 5880 raft_consensus.cc:729] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 847d2979047844d7a9bbbaee2165e163, State: Initialized, Role: FOLLOWER
I20260620 06:02:03.566002 5879 raft_consensus.cc:348] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.566126 5879 raft_consensus.cc:374] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:03.566107 5880 consensus_queue.cc:260] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.566170 5879 raft_consensus.cc:729] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edfeac70e44e4bf0871fff45cf2eb464, State: Initialized, Role: FOLLOWER
I20260620 06:02:03.566306 5879 consensus_queue.cc:260] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.566380 5880 ts_tablet_manager.cc:1428] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:03.566596 5879 ts_tablet_manager.cc:1428] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:02:03.566653 5881 tablet_bootstrap.cc:492] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Bootstrap starting.
I20260620 06:02:03.567508 5881 tablet_bootstrap.cc:654] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:03.568837 5881 tablet_bootstrap.cc:492] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: No bootstrap required, opened a new log
I20260620 06:02:03.568933 5881 ts_tablet_manager.cc:1397] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:03.569343 5881 raft_consensus.cc:348] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.569487 5881 raft_consensus.cc:374] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:03.569533 5881 raft_consensus.cc:729] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Initialized, Role: FOLLOWER
I20260620 06:02:03.569666 5881 consensus_queue.cc:260] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.569952 5881 ts_tablet_manager.cc:1428] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:03.660037 5853 raft_consensus.cc:482] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:03.660209 5853 raft_consensus.cc:504] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.660694 5853 leader_election.cc:288] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353), 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:03.661170 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3fd6fd469442288f159d9ecf8f413c" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfeac70e44e4bf0871fff45cf2eb464" is_pre_election: true
I20260620 06:02:03.661181 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3fd6fd469442288f159d9ecf8f413c" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163" is_pre_election: true
I20260620 06:02:03.661334 5438 raft_consensus.cc:2454] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 0.
I20260620 06:02:03.661336 5578 raft_consensus.cc:2454] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 0.
I20260620 06:02:03.661676 5652 leader_election.cc:302] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, 847d2979047844d7a9bbbaee2165e163; no voters:
I20260620 06:02:03.661898 5853 raft_consensus.cc:2781] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:03.661969 5853 raft_consensus.cc:482] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:03.662011 5853 raft_consensus.cc:3037] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:03.663134 5853 raft_consensus.cc:504] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.663446 5853 leader_election.cc:288] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 election: Requested vote from peers edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353), 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281)
I20260620 06:02:03.663851 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3fd6fd469442288f159d9ecf8f413c" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "847d2979047844d7a9bbbaee2165e163"
I20260620 06:02:03.663873 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce3fd6fd469442288f159d9ecf8f413c" candidate_uuid: "3239af37a9584403b4bcffa0756a6a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edfeac70e44e4bf0871fff45cf2eb464"
I20260620 06:02:03.663995 5578 raft_consensus.cc:3037] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:03.664016 5438 raft_consensus.cc:3037] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:03.664994 5578 raft_consensus.cc:2454] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 1.
I20260620 06:02:03.665087 5438 raft_consensus.cc:2454] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3239af37a9584403b4bcffa0756a6a0c in term 1.
I20260620 06:02:03.665355 5652 leader_election.cc:302] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3239af37a9584403b4bcffa0756a6a0c, 847d2979047844d7a9bbbaee2165e163; no voters:
I20260620 06:02:03.665539 5853 raft_consensus.cc:2781] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:03.665637 5853 raft_consensus.cc:686] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 1 LEADER]: Becoming Leader. State: Replica: 3239af37a9584403b4bcffa0756a6a0c, State: Running, Role: LEADER
I20260620 06:02:03.665784 5853 consensus_queue.cc:237] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } }
I20260620 06:02:03.667397 5280 catalog_manager.cc:5495] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3). New cstate: current_term: 1 leader_uuid: "3239af37a9584403b4bcffa0756a6a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3239af37a9584403b4bcffa0756a6a0c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 34429 } health_report { overall_health: HEALTHY } } }
I20260620 06:02:03.699307 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:03.695754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.3.115.3:34429)
0620 06:02:03.696090 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:03.696106 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:03.696117 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:02:03.696154 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:03.696156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:03.696197 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:03.696290 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:03.696542 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.697239 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.698176 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.698518 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.698618 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:03.698813 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:03.698895 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:03.698969 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:03.699097 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 06:02:03.699151 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":111,"threads_started":1}
I20260620 06:02:03.709208 5896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:03.705785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:55813 (local address 127.3.115.3:34429)
0620 06:02:03.705902 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:03.705907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:03.706071 (+ 164us) server_negotiation.cc:407] Connection header received
0620 06:02:03.706156 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:03.706159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:03.706205 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:03.706281 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:03.706519 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.707262 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.708087 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.708412 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.708512 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:03.708783 (+ 271us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:03.708858 (+ 75us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:03.708929 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:03.709003 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 06:02:03.709055 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":42}
I20260620 06:02:03.709206 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:03.705689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:34429 (local address 127.3.115.3:55813)
0620 06:02:03.705993 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:03.706005 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:03.706088 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:03.706298 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:03.706300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:03.706306 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:03.706419 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:03.706423 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.707368 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.707371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:03.707973 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:03.707982 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:03.708518 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:03.708521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:03.708596 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:03.708620 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:03.708939 (+ 319us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:03.708944 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:02:03.708995 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 06:02:03.709062 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":18,"thread_start_us":96,"threads_started":1}
I20260620 06:02:03.715049 5438 raft_consensus.cc:1261] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464 [term 1 FOLLOWER]: Refusing update from remote peer 3239af37a9584403b4bcffa0756a6a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:03.715138 5578 raft_consensus.cc:1261] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Refusing update from remote peer 3239af37a9584403b4bcffa0756a6a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:03.715457 5848 consensus_queue.cc:1035] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "edfeac70e44e4bf0871fff45cf2eb464" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:03.715626 5844 consensus_queue.cc:1035] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "847d2979047844d7a9bbbaee2165e163" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:02:03.730489 5846 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 06:02:05.302012 5280 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:05.302210 5280 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55584:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:05.304908 5280 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e6f40964de4480c93fd18eb0f134f0a: Sending DeleteTablet for 3 replicas of tablet ce3fd6fd469442288f159d9ecf8f413c
I20260620 06:02:05.305505 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3fd6fd469442288f159d9ecf8f413c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:05 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:36554
I20260620 06:02:05.305649 5543 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3fd6fd469442288f159d9ecf8f413c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:05 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:55076
I20260620 06:02:05.305950 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 5353
I20260620 06:02:05.305998 5922 tablet_replica.cc:330] T ce3fd6fd469442288f159d9ecf8f413c P edfeac70e44e4bf0871fff45cf2eb464: stopping tablet replica
I20260620 06:02:05.307324 5923 tablet_replica.cc:330] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: stopping tablet replica
I20260620 06:02:05.307502 5923 raft_consensus.cc:2229] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:05.307659 5923 raft_consensus.cc:2258] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:05.308301 5923 ts_tablet_manager.cc:1905] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:05.308851 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet ce3fd6fd469442288f159d9ecf8f413c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:05 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:34144
I20260620 06:02:05.309275 5924 tablet_replica.cc:330] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: stopping tablet replica
I20260620 06:02:05.309448 5924 raft_consensus.cc:2229] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:05.309880 5924 raft_consensus.cc:2258] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:05.310618 5924 ts_tablet_manager.cc:1905] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:05.311300 5923 ts_tablet_manager.cc:1918] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:02:05.311388 5923 log.cc:1196] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/ce3fd6fd469442288f159d9ecf8f413c
I20260620 06:02:05.311753 5923 ts_tablet_manager.cc:1939] T ce3fd6fd469442288f159d9ecf8f413c P 847d2979047844d7a9bbbaee2165e163: Deleting consensus metadata
I20260620 06:02:05.312544 5268 catalog_manager.cc:4841] TS 847d2979047844d7a9bbbaee2165e163 (127.3.115.2:33281): tablet ce3fd6fd469442288f159d9ecf8f413c (table test-table [id=bfb5edd29bdd4bbaac27c113ee69434a]) successfully deleted
I20260620 06:02:05.313732 5924 ts_tablet_manager.cc:1918] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 06:02:05.313843 5924 log.cc:1196] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/ce3fd6fd469442288f159d9ecf8f413c
I20260620 06:02:05.314158 5924 ts_tablet_manager.cc:1939] T ce3fd6fd469442288f159d9ecf8f413c P 3239af37a9584403b4bcffa0756a6a0c: Deleting consensus metadata
I20260620 06:02:05.314967 5265 catalog_manager.cc:4841] TS 3239af37a9584403b4bcffa0756a6a0c (127.3.115.3:34429): tablet ce3fd6fd469442288f159d9ecf8f413c (table test-table [id=bfb5edd29bdd4bbaac27c113ee69434a]) successfully deleted
W20260620 06:02:05.315802 5265 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:35353 (error 108)
I20260620 06:02:05.315992 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 5493
I20260620 06:02:05.317721 5926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:05.317207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35353 (local address 127.0.0.1:34882)
0620 06:02:05.317483 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:05.317580 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35353: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
W20260620 06:02:05.317893 5265 catalog_manager.cc:4584] TS edfeac70e44e4bf0871fff45cf2eb464 (127.3.115.1:35353): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ce3fd6fd469442288f159d9ecf8f413c: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35353: connect: Connection refused (error 111)
I20260620 06:02:05.323661 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 5633
I20260620 06:02:05.330845 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 5249
2026-06-20T06:02:05Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6276 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:05 dist-test-slave-4h14 krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:05 dist-test-slave-4h14 krb5kdc[5929](info): set up 2 sockets
Jun 20 06:02:05 dist-test-slave-4h14 krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260620 06:02:06.369314 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:06 dist-test-slave-4h14 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:06.489753 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35627
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:06.604781 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:06.605132 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:06.605211 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:06.609578 5942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:06.609655 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:06.609675 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:06.609692 5942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:06.609707 5942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:06.615413 5942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:06.616617 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:06.617766 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:06.627163 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:06.627146 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:06.627236 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:06.627439 5942 server_base.cc:1034] running on GCE node
I20260620 06:02:06.628075 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:06.629347 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:06.630555 5942 hybrid_clock.cc:648] HybridClock initialized: now 1781935326630543 us; error 31 us; skew 500 ppm
Jun 20 06:02:06 dist-test-slave-4h14 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:06.633522 5942 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:06.634569 5942 webserver.cc:460] Webserver started at http://127.3.115.62:34949/ using document root <none> and password file <none>
I20260620 06:02:06.635186 5942 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:06.635285 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:06.635586 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:06.637930 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "9490f979b5954e0482feb941d177aac6"
format_stamp: "Formatted at 2026-06-20 06:02:06 on dist-test-slave-4h14"
server_key: "8a1c7a983371759b2e908ff9cb839fe3"
server_key_iv: "ebc070ce6f327158e28adf7d8cd93d15"
server_key_version: "encryptionkey@0"
I20260620 06:02:06.638541 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "9490f979b5954e0482feb941d177aac6"
format_stamp: "Formatted at 2026-06-20 06:02:06 on dist-test-slave-4h14"
server_key: "8a1c7a983371759b2e908ff9cb839fe3"
server_key_iv: "ebc070ce6f327158e28adf7d8cd93d15"
server_key_version: "encryptionkey@0"
I20260620 06:02:06.642621 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:06.645288 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:06.646641 5942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:06.646795 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "9490f979b5954e0482feb941d177aac6"
format_stamp: "Formatted at 2026-06-20 06:02:06 on dist-test-slave-4h14"
server_key: "8a1c7a983371759b2e908ff9cb839fe3"
server_key_iv: "ebc070ce6f327158e28adf7d8cd93d15"
server_key_version: "encryptionkey@0"
I20260620 06:02:06.646920 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:06.665132 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:06.668701 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:06.668960 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:06.676625 5942 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:35627
I20260620 06:02:06.676646 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:35627 every 8 connection(s)
I20260620 06:02:06.677670 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:06.680495 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:06.686147 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6: Bootstrap starting.
I20260620 06:02:06.686468 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 5942
I20260620 06:02:06.686559 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:06.686830 3532 external_mini_cluster.cc:1428] Setting key a03650b2195b5fb104baa5d3e1a9b5c9
I20260620 06:02:06.689235 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:06.690006 6010 log.cc:824] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:06.692159 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6: No bootstrap required, opened a new log
I20260620 06:02:06.695683 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9490f979b5954e0482feb941d177aac6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35627 } }
I20260620 06:02:06.695950 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:06.696033 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9490f979b5954e0482feb941d177aac6, State: Initialized, Role: FOLLOWER
I20260620 06:02:06.696563 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9490f979b5954e0482feb941d177aac6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35627 } }
I20260620 06:02:06.696767 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:06.696838 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:06.696933 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 06:02:06 dist-test-slave-4h14 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:06.697887 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9490f979b5954e0482feb941d177aac6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35627 } }
I20260620 06:02:06.698240 6010 leader_election.cc:302] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9490f979b5954e0482feb941d177aac6; no voters:
I20260620 06:02:06.698603 6010 leader_election.cc:288] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:06.698716 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:06.698964 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [term 1 LEADER]: Becoming Leader. State: Replica: 9490f979b5954e0482feb941d177aac6, State: Running, Role: LEADER
I20260620 06:02:06.699277 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9490f979b5954e0482feb941d177aac6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35627 } }
I20260620 06:02:06.700029 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:06.701099 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9490f979b5954e0482feb941d177aac6. Latest consensus state: current_term: 1 leader_uuid: "9490f979b5954e0482feb941d177aac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9490f979b5954e0482feb941d177aac6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35627 } } }
I20260620 06:02:06.701228 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:06.701520 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9490f979b5954e0482feb941d177aac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9490f979b5954e0482feb941d177aac6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35627 } } }
I20260620 06:02:06.701637 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:06.701642 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:06.704373 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:06.688202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.3.115.62:35627)
0620 06:02:06.688641 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:06.688701 (+ 60us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:06.688748 (+ 47us) server_negotiation.cc:407] Connection header received
0620 06:02:06.689556 (+ 808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:06.689583 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:06.690090 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:06.690505 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:06.691433 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:06.693177 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:06.694046 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:06.694565 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:06.694732 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:06.697579 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:06.697590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:06.697604 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:06.697632 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:06.699756 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:06.700305 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:06.700311 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:06.700317 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:06.700470 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:06.700845 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:06.700851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:06.700855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:06.701586 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:06.702503 (+ 917us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:06.703119 (+ 616us) server_negotiation.cc:299] Negotiation successful
0620 06:02:06.703428 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":296,"thread_start_us":146,"threads_started":1}
I20260620 06:02:06.705098 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:06.711108 6024 catalog_manager.cc:1347] Generated new cluster ID: fef815c9ef59476b8d86cb266913d2b1
I20260620 06:02:06.711189 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:06.721971 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:06.723120 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:06.730031 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9490f979b5954e0482feb941d177aac6: Generated new TSK 0
I20260620 06:02:06.730775 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:06.760145 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35627
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:06.875489 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:06.875774 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:06.875819 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:06.879914 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:06.879997 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:06.880060 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:06.885392 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:06.886519 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:06.887527 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:06.897148 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:06.897171 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:06.897161 6036 server_base.cc:1034] running on GCE node
W20260620 06:02:06.897178 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:06.897801 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:06.898437 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:06.899650 6036 hybrid_clock.cc:648] HybridClock initialized: now 1781935326899633 us; error 37 us; skew 500 ppm
Jun 20 06:02:06 dist-test-slave-4h14 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:06.902877 6036 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:06.903800 6036 webserver.cc:460] Webserver started at http://127.3.115.1:38791/ using document root <none> and password file <none>
I20260620 06:02:06.904354 6036 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:06.904414 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:06.904589 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:06.906909 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "adde10244c80458592818af3b2c43528"
format_stamp: "Formatted at 2026-06-20 06:02:06 on dist-test-slave-4h14"
server_key: "24a989f6505e6661dd6b30f8a0d4b61a"
server_key_iv: "bdcc2ea66125d7806d5d1ba65f6985a4"
server_key_version: "encryptionkey@0"
I20260620 06:02:06.907471 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "adde10244c80458592818af3b2c43528"
format_stamp: "Formatted at 2026-06-20 06:02:06 on dist-test-slave-4h14"
server_key: "24a989f6505e6661dd6b30f8a0d4b61a"
server_key_iv: "bdcc2ea66125d7806d5d1ba65f6985a4"
server_key_version: "encryptionkey@0"
I20260620 06:02:06.911388 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 06:02:06.913985 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:06.915350 6036 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:06.915505 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "adde10244c80458592818af3b2c43528"
format_stamp: "Formatted at 2026-06-20 06:02:06 on dist-test-slave-4h14"
server_key: "24a989f6505e6661dd6b30f8a0d4b61a"
server_key_iv: "bdcc2ea66125d7806d5d1ba65f6985a4"
server_key_version: "encryptionkey@0"
I20260620 06:02:06.915629 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:06.929613 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:06.932789 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:06.933041 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:06.933665 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:06.934495 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:06.934540 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:06.934610 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:06.934643 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:06.943617 6036 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:33191
I20260620 06:02:06.943638 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:33191 every 8 connection(s)
I20260620 06:02:06.944639 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:06.947016 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6036
I20260620 06:02:06.947145 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:06.947512 3532 external_mini_cluster.cc:1428] Setting key 0e83a3dc7a744c4bf7411ad28afe9c30
Jun 20 06:02:06 dist-test-slave-4h14 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935326, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:06.960083 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:06.946500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:33053 (local address 127.3.115.62:35627)
0620 06:02:06.946632 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:06.946635 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:06.947786 (+ 1151us) server_negotiation.cc:407] Connection header received
0620 06:02:06.949329 (+ 1543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:06.949332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:06.949374 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:06.949434 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:06.951096 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:06.952008 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:06.953298 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:06.953719 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:06.953844 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:06.956338 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:06.956345 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:06.956349 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:06.956382 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:06.958006 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:06.958531 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:06.958534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:06.958535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:06.958611 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:06.958909 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:06.958911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:06.958912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:06.959085 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:06.959160 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:06.959709 (+ 549us) server_negotiation.cc:299] Negotiation successful
0620 06:02:06.959879 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:02:06.961170 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:06.947022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35627 (local address 127.3.115.1:33053)
0620 06:02:06.947597 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:06.947660 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:06.949100 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:06.949650 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:06.949661 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:06.950224 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:06.950937 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:06.950954 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:06.952131 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:06.952134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:06.953148 (+ 1014us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:06.953162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:06.953885 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:06.953888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:06.954078 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:06.954704 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:06.954722 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:06.956161 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:06.958109 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:06.958114 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:06.958128 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:06.958430 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:06.958692 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:06.958694 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:06.958696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:06.958830 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:06.959172 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:06.959177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:06.959482 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 06:02:06.959661 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 06:02:06.959998 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":169,"threads_started":1}
I20260620 06:02:06.962837 6165 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35627
I20260620 06:02:06.963106 6165 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:06.963617 6165 heartbeater.cc:511] Master 127.3.115.62:35627 requested a full tablet report, sending...
I20260620 06:02:06.965197 5974 ts_manager.cc:194] Registered new tserver with Master: adde10244c80458592818af3b2c43528 (127.3.115.1:33191)
I20260620 06:02:06.966382 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:33053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:06.976138 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35627
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:07.091207 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:07.091471 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:07.091514 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:07.095590 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:07.095649 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:07.095707 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:07.100947 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:07.102061 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:07.103097 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:07.112726 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:07.112821 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:07.112862 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:07.112970 6170 server_base.cc:1034] running on GCE node
I20260620 06:02:07.113541 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:07.114151 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:07.115307 6170 hybrid_clock.cc:648] HybridClock initialized: now 1781935327115286 us; error 39 us; skew 500 ppm
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:07.118157 6170 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:07.119050 6170 webserver.cc:460] Webserver started at http://127.3.115.2:35025/ using document root <none> and password file <none>
I20260620 06:02:07.119638 6170 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:07.119720 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:07.119935 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:07.122026 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "b6deda05facf4375b44600b8d5af5f73"
format_stamp: "Formatted at 2026-06-20 06:02:07 on dist-test-slave-4h14"
server_key: "d24fc16ddc01d40a265948e2a85f800f"
server_key_iv: "f4bb20be8198de25fce5d4217ff0e13b"
server_key_version: "encryptionkey@0"
I20260620 06:02:07.122582 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "b6deda05facf4375b44600b8d5af5f73"
format_stamp: "Formatted at 2026-06-20 06:02:07 on dist-test-slave-4h14"
server_key: "d24fc16ddc01d40a265948e2a85f800f"
server_key_iv: "f4bb20be8198de25fce5d4217ff0e13b"
server_key_version: "encryptionkey@0"
I20260620 06:02:07.126407 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:07.128909 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:07.130499 6170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:07.130630 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "b6deda05facf4375b44600b8d5af5f73"
format_stamp: "Formatted at 2026-06-20 06:02:07 on dist-test-slave-4h14"
server_key: "d24fc16ddc01d40a265948e2a85f800f"
server_key_iv: "f4bb20be8198de25fce5d4217ff0e13b"
server_key_version: "encryptionkey@0"
I20260620 06:02:07.130745 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:07.150285 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:07.153954 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:07.154183 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:07.154835 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:07.155694 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:07.155768 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:07.155843 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:07.155885 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:07.165131 6170 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:44447
I20260620 06:02:07.165151 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:44447 every 8 connection(s)
I20260620 06:02:07.166069 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:07.173305 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6170
I20260620 06:02:07.173421 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:07.173784 3532 external_mini_cluster.cc:1428] Setting key f865eb47f62bfe200c7362c88275aa25
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:07.180397 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:07.167943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:40065 (local address 127.3.115.62:35627)
0620 06:02:07.168105 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:07.168109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:07.169032 (+ 923us) server_negotiation.cc:407] Connection header received
0620 06:02:07.170365 (+ 1333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:07.170369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:07.170423 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:07.170545 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:07.171766 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.172289 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.173648 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.173907 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.174047 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:07.176750 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:07.176753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:07.176756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:07.176788 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:07.178280 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:07.178821 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:07.178829 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:07.178830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:07.178910 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:07.179282 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:07.179286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:07.179287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:07.179471 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:07.179575 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:07.180097 (+ 522us) server_negotiation.cc:299] Negotiation successful
0620 06:02:07.180226 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 06:02:07.181308 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:07.168345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35627 (local address 127.3.115.2:40065)
0620 06:02:07.168857 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:07.168902 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:07.170157 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:07.170719 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:07.170727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:07.171088 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:07.171619 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:07.171632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.172415 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.172421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:07.173438 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:07.173450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.174092 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.174096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:07.174309 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:07.174980 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:07.174999 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:07.176553 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:07.178411 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:07.178418 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:07.178435 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:07.178708 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:07.179031 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:07.179034 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:07.179036 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:07.179182 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:07.179584 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:07.179593 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:07.179851 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 06:02:07.180043 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 06:02:07.180381 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":127,"threads_started":1}
I20260620 06:02:07.182519 6299 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35627
I20260620 06:02:07.182796 6299 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:02:07.183396 6299 heartbeater.cc:511] Master 127.3.115.62:35627 requested a full tablet report, sending...
I20260620 06:02:07.184510 5974 ts_manager.cc:194] Registered new tserver with Master: b6deda05facf4375b44600b8d5af5f73 (127.3.115.2:44447)
I20260620 06:02:07.185206 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:40065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:07.201400 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35627
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:07.316256 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:07.316565 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:07.316640 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:07.320909 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:07.320998 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:07.321090 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:07.326440 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:07.327589 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:07.328766 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:07.338717 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:07.338712 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:07.338894 6304 server_base.cc:1034] running on GCE node
W20260620 06:02:07.339191 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:07.339643 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:07.340369 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:07.341562 6304 hybrid_clock.cc:648] HybridClock initialized: now 1781935327341522 us; error 54 us; skew 500 ppm
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:07.344426 6304 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:07.345346 6304 webserver.cc:460] Webserver started at http://127.3.115.3:41247/ using document root <none> and password file <none>
I20260620 06:02:07.345896 6304 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:07.345958 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:07.346127 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:07.348253 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "07aa14f1dfac45778360693b3880004f"
format_stamp: "Formatted at 2026-06-20 06:02:07 on dist-test-slave-4h14"
server_key: "c98d32d3212d4f12acf4ba29617ed3bb"
server_key_iv: "7179a910269f2a4f1ccadd684ff8b936"
server_key_version: "encryptionkey@0"
I20260620 06:02:07.348867 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "07aa14f1dfac45778360693b3880004f"
format_stamp: "Formatted at 2026-06-20 06:02:07 on dist-test-slave-4h14"
server_key: "c98d32d3212d4f12acf4ba29617ed3bb"
server_key_iv: "7179a910269f2a4f1ccadd684ff8b936"
server_key_version: "encryptionkey@0"
I20260620 06:02:07.353060 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:07.355698 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:07.357208 6304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:07.357334 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "07aa14f1dfac45778360693b3880004f"
format_stamp: "Formatted at 2026-06-20 06:02:07 on dist-test-slave-4h14"
server_key: "c98d32d3212d4f12acf4ba29617ed3bb"
server_key_iv: "7179a910269f2a4f1ccadd684ff8b936"
server_key_version: "encryptionkey@0"
I20260620 06:02:07.357452 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:07.375625 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:07.378975 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:07.379179 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:07.379779 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:07.380611 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:07.380719 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:07.380790 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:07.380844 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:07.390334 6304 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40237
I20260620 06:02:07.390367 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40237 every 8 connection(s)
I20260620 06:02:07.391299 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:07.397811 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6304
I20260620 06:02:07.397943 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:07.398253 3532 external_mini_cluster.cc:1428] Setting key e3a718f90b07653886de90034b54f991
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935327, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:07.405920 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:07.393216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:58503 (local address 127.3.115.62:35627)
0620 06:02:07.393360 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:07.393364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:07.394222 (+ 858us) server_negotiation.cc:407] Connection header received
0620 06:02:07.395586 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:07.395589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:07.395645 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:07.395760 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:07.396985 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.397514 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.398899 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.399158 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.399272 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:07.402230 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:07.402233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:07.402235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:07.402261 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:07.403779 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:07.404372 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:07.404374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:07.404375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:07.404451 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:07.404840 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:07.404842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:07.404844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:07.405040 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:07.405119 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:07.405633 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 06:02:07.405763 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:02:07.406895 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:07.393620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35627 (local address 127.3.115.3:58503)
0620 06:02:07.394065 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:07.394109 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:07.395350 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:07.395932 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:07.395939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:07.396308 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:07.396827 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:07.396841 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.397642 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.397646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:07.398784 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:07.398794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:07.399342 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:07.399347 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:07.399595 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:07.400285 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:07.400302 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:07.402056 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:07.403932 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:07.403941 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:07.403963 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:07.404283 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:07.404538 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:07.404540 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:07.404542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:07.404754 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:07.405134 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:07.405139 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:07.405391 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 06:02:07.405565 (+ 174us) client_negotiation.cc:240] Negotiation successful
0620 06:02:07.405901 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":113,"threads_started":1}
I20260620 06:02:07.408108 6433 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35627
I20260620 06:02:07.408372 6433 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:07.408947 6433 heartbeater.cc:511] Master 127.3.115.62:35627 requested a full tablet report, sending...
I20260620 06:02:07.410004 5974 ts_manager.cc:194] Registered new tserver with Master: 07aa14f1dfac45778360693b3880004f (127.3.115.3:40237)
I20260620 06:02:07.410590 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:58503
I20260620 06:02:07.412781 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:02:07.426249 6013 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:07.424753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.3.115.62:35627)
0620 06:02:07.424902 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:07.424906 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:07.425104 (+ 198us) server_negotiation.cc:407] Connection header received
0620 06:02:07.425211 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:07.425213 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:07.425259 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:07.425331 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:07.426094 (+ 763us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41860: BlockingRecv error: recv got EOF from 127.0.0.1:41860 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:02:07.427265 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6036
I20260620 06:02:07.432809 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6170
I20260620 06:02:07.438433 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6304
I20260620 06:02:07.444023 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 5942
2026-06-20T06:02:07Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2110 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[6445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[6445](info): set up 2 sockets
Jun 20 06:02:07 dist-test-slave-4h14 krb5kdc[6445](info): commencing operation
krb5kdc: starting...
W20260620 06:02:08.497231 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:08 dist-test-slave-4h14 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:08.623255 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40095
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:35631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:40095
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:08.737452 6458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:08.737794 6458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:08.737869 6458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:08.741989 6458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:08.742101 6458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:08.742139 6458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:08.742180 6458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:08.742210 6458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:08.747624 6458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:40095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:08.748806 6458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:08.749897 6458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:08.758644 6466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:08.758663 6463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:08.758656 6464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:08.759400 6458 server_base.cc:1034] running on GCE node
I20260620 06:02:08.760164 6458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:08.761871 6458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:08.763101 6458 hybrid_clock.cc:648] HybridClock initialized: now 1781935328763062 us; error 51 us; skew 500 ppm
Jun 20 06:02:08 dist-test-slave-4h14 krb5kdc[6445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935328, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:08.770231 6458 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:08.771327 6458 webserver.cc:460] Webserver started at http://127.3.115.62:44009/ using document root <none> and password file <none>
I20260620 06:02:08.771968 6458 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:08.772071 6458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:08.772325 6458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:08.774677 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "61409b26af464bf8846f5df7638ec5c0"
format_stamp: "Formatted at 2026-06-20 06:02:08 on dist-test-slave-4h14"
server_key: "6e607b4b6493f9cb07ff142ed1b7a40a"
server_key_iv: "640e4a9d26e268e632af686ef825437c"
server_key_version: "encryptionkey@0"
I20260620 06:02:08.775331 6458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "61409b26af464bf8846f5df7638ec5c0"
format_stamp: "Formatted at 2026-06-20 06:02:08 on dist-test-slave-4h14"
server_key: "6e607b4b6493f9cb07ff142ed1b7a40a"
server_key_iv: "640e4a9d26e268e632af686ef825437c"
server_key_version: "encryptionkey@0"
I20260620 06:02:08.779459 6458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:08.782258 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:08.783625 6458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:02:08.783738 6458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "61409b26af464bf8846f5df7638ec5c0"
format_stamp: "Formatted at 2026-06-20 06:02:08 on dist-test-slave-4h14"
server_key: "6e607b4b6493f9cb07ff142ed1b7a40a"
server_key_iv: "640e4a9d26e268e632af686ef825437c"
server_key_version: "encryptionkey@0"
I20260620 06:02:08.783859 6458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:08.799225 6458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:08.802505 6458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:08.802693 6458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:08.809952 6458 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:40095
I20260620 06:02:08.810261 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:40095 every 8 connection(s)
I20260620 06:02:08.810985 6458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:08.812238 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6458
I20260620 06:02:08.812355 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:08.812703 3532 external_mini_cluster.cc:1428] Setting key 444a51614eb9d3e12dd53e04fb9d8e20
I20260620 06:02:08.814409 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:08.821069 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0: Bootstrap starting.
Jun 20 06:02:08 dist-test-slave-4h14 krb5kdc[6445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:08.823824 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:08.824792 6526 log.cc:824] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:08.827253 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0: No bootstrap required, opened a new log
I20260620 06:02:08.827980 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:08.814034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.3.115.62:40095)
0620 06:02:08.814473 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:08.814493 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:08.814535 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:02:08.815210 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:08.815227 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:08.815555 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:08.815870 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:08.816857 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:08.817876 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:08.819047 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:08.819519 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:08.819644 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:08.822458 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:08.822465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:08.822481 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:08.822522 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:08.824717 (+ 2195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:08.825238 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:08.825245 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:08.825251 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:08.825372 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:08.825704 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:08.825707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:08.825708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:08.826204 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:08.826396 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:08.826673 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 06:02:08.826983 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":162,"threads_started":1}
I20260620 06:02:08.830914 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:02:08.831672 6526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61409b26af464bf8846f5df7638ec5c0" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40095 } }
I20260620 06:02:08.831929 6526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:08.832019 6526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61409b26af464bf8846f5df7638ec5c0, State: Initialized, Role: FOLLOWER
I20260620 06:02:08.832459 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61409b26af464bf8846f5df7638ec5c0" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40095 } }
I20260620 06:02:08.832621 6526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:08.832755 6526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:08.832881 6526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:08.834314 6526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61409b26af464bf8846f5df7638ec5c0" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40095 } }
I20260620 06:02:08.834712 6526 leader_election.cc:302] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61409b26af464bf8846f5df7638ec5c0; no voters:
I20260620 06:02:08.835098 6526 leader_election.cc:288] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:08.835255 6532 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:08.835538 6532 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [term 1 LEADER]: Becoming Leader. State: Replica: 61409b26af464bf8846f5df7638ec5c0, State: Running, Role: LEADER
I20260620 06:02:08.835888 6532 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61409b26af464bf8846f5df7638ec5c0" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40095 } }
I20260620 06:02:08.836591 6526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:08.837288 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61409b26af464bf8846f5df7638ec5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61409b26af464bf8846f5df7638ec5c0" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40095 } } }
I20260620 06:02:08.837507 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:08.837811 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61409b26af464bf8846f5df7638ec5c0. Latest consensus state: current_term: 1 leader_uuid: "61409b26af464bf8846f5df7638ec5c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61409b26af464bf8846f5df7638ec5c0" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40095 } } }
I20260620 06:02:08.837920 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:08.838205 6542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:08.841552 6542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:08.848389 6542 catalog_manager.cc:1347] Generated new cluster ID: f14c708e4a2b41648597bdc75475db4f
I20260620 06:02:08.848505 6542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:08.854211 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:08.847745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.3.115.62:40095)
0620 06:02:08.847888 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:08.847893 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:08.848797 (+ 904us) server_negotiation.cc:407] Connection header received
0620 06:02:08.848966 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:08.848970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:08.849025 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:08.849129 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:08.849770 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:08.850567 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:08.851785 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:08.852142 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:08.852233 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:08.853230 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:08.853233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:08.853236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:08.853760 (+ 524us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:08.853790 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:08.853807 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:08.853892 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:08.854016 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 06:02:08.854072 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:02:08.882335 6542 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:08.883358 6542 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:08.890038 6542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61409b26af464bf8846f5df7638ec5c0: Generated new TSK 0
I20260620 06:02:08.890738 6542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:08.900586 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6458
2026-06-20T06:02:08Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1454 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:08 dist-test-slave-4h14 krb5kdc[6559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:08 dist-test-slave-4h14 krb5kdc[6559](info): set up 2 sockets
Jun 20 06:02:08 dist-test-slave-4h14 krb5kdc[6559](info): commencing operation
krb5kdc: starting...
W20260620 06:02:09.961301 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:10.088583 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35215
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:35215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:10.201463 6572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:10.201783 6572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:10.201851 6572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:10.205977 6572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:10.206073 6572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:10.206111 6572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:10.206148 6572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:10.206183 6572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:10.211490 6572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:35215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:10.212623 6572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:10.213702 6572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:10.222002 6577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.222002 6578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.222011 6580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:10.222512 6572 server_base.cc:1034] running on GCE node
I20260620 06:02:10.223076 6572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:10.224228 6572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:10.225437 6572 hybrid_clock.cc:648] HybridClock initialized: now 1781935330225415 us; error 33 us; skew 500 ppm
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:10.228353 6572 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:10.229334 6572 webserver.cc:460] Webserver started at http://127.3.115.62:41145/ using document root <none> and password file <none>
I20260620 06:02:10.229898 6572 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:10.229991 6572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:10.230226 6572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:10.232276 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "34070182bc82428d9b68bf0fc0afcc16"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "43a5b5859ec432f7bd4186115d907240"
server_key_iv: "36b5471055e16dcc32dfaf6bdb92358b"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.232892 6572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "34070182bc82428d9b68bf0fc0afcc16"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "43a5b5859ec432f7bd4186115d907240"
server_key_iv: "36b5471055e16dcc32dfaf6bdb92358b"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.236699 6572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:02:10.239125 6587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.240382 6572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:10.240526 6572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "34070182bc82428d9b68bf0fc0afcc16"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "43a5b5859ec432f7bd4186115d907240"
server_key_iv: "36b5471055e16dcc32dfaf6bdb92358b"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.240643 6572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:10.254921 6572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:10.257908 6572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:10.258141 6572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:10.265697 6572 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:35215
I20260620 06:02:10.265724 6639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:35215 every 8 connection(s)
I20260620 06:02:10.266642 6572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:10.269342 6640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:10.274739 6640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16: Bootstrap starting.
I20260620 06:02:10.275439 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6572
I20260620 06:02:10.275529 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:10.275791 3532 external_mini_cluster.cc:1428] Setting key 698f9fafb4ee18dd976bac3b77ba586a
I20260620 06:02:10.277510 6640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:10.278384 6640 log.cc:824] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:10.280445 6640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16: No bootstrap required, opened a new log
I20260620 06:02:10.284546 6640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070182bc82428d9b68bf0fc0afcc16" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35215 } }
I20260620 06:02:10.284853 6640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:10.284921 6640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34070182bc82428d9b68bf0fc0afcc16, State: Initialized, Role: FOLLOWER
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:10.285437 6640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070182bc82428d9b68bf0fc0afcc16" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35215 } }
I20260620 06:02:10.285609 6640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:10.285683 6640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:10.285794 6640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:10.286932 6640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070182bc82428d9b68bf0fc0afcc16" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35215 } }
I20260620 06:02:10.287309 6640 leader_election.cc:302] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34070182bc82428d9b68bf0fc0afcc16; no voters:
I20260620 06:02:10.287645 6640 leader_election.cc:288] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:10.287763 6645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:10.287990 6645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [term 1 LEADER]: Becoming Leader. State: Replica: 34070182bc82428d9b68bf0fc0afcc16, State: Running, Role: LEADER
I20260620 06:02:10.288388 6645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070182bc82428d9b68bf0fc0afcc16" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35215 } }
I20260620 06:02:10.289011 6640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:10.290876 6646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34070182bc82428d9b68bf0fc0afcc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070182bc82428d9b68bf0fc0afcc16" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35215 } } }
I20260620 06:02:10.291011 6646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:10.291304 6647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34070182bc82428d9b68bf0fc0afcc16. Latest consensus state: current_term: 1 leader_uuid: "34070182bc82428d9b68bf0fc0afcc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34070182bc82428d9b68bf0fc0afcc16" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35215 } } }
I20260620 06:02:10.291397 6659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:10.291606 6647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:10.291687 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:10.277070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.3.115.62:35215)
0620 06:02:10.277541 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:10.277560 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:10.277603 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:02:10.278474 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:10.278508 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:10.278901 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:10.279253 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:10.279892 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.281371 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.282228 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.282958 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.283136 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:10.285931 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:10.285937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:10.285954 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:10.285992 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:10.287983 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:10.288554 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:10.288559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:10.288566 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:10.288732 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:10.289059 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:10.289062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:10.289063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:10.289568 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:10.289771 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:10.290117 (+ 346us) server_negotiation.cc:299] Negotiation successful
0620 06:02:10.290445 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":110,"threads_started":1}
I20260620 06:02:10.294481 6659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:10.299515 6659 catalog_manager.cc:1347] Generated new cluster ID: 2fba963c67be464b89c4c0aae791e13b
I20260620 06:02:10.299587 6659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:10.311008 6659 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:10.312301 6659 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:10.319382 6659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34070182bc82428d9b68bf0fc0afcc16: Generated new TSK 0
I20260620 06:02:10.320259 6659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:10.348284 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35215
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:10.463004 6666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:10.463279 6666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:10.463322 6666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:10.467468 6666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:10.467535 6666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:10.467638 6666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:10.472934 6666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:10.474094 6666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:10.475129 6666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:10.484572 6671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:10.484735 6666 server_base.cc:1034] running on GCE node
W20260620 06:02:10.484616 6674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.484601 6672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:10.485224 6666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:10.485858 6666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:10.487053 6666 hybrid_clock.cc:648] HybridClock initialized: now 1781935330487032 us; error 30 us; skew 500 ppm
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:10.489874 6666 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:10.490927 6666 webserver.cc:460] Webserver started at http://127.3.115.1:37075/ using document root <none> and password file <none>
I20260620 06:02:10.491582 6666 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:10.491680 6666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:10.491900 6666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:10.494177 6666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "87a05578210249cb85bbe517c0de92c4"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "9cca23890d7f10c71f191d0c08b18362"
server_key_iv: "c46d7c53bbefe702f5322d53193467c0"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.494794 6666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "87a05578210249cb85bbe517c0de92c4"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "9cca23890d7f10c71f191d0c08b18362"
server_key_iv: "c46d7c53bbefe702f5322d53193467c0"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.498780 6666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:10.501394 6681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.502647 6666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:10.502794 6666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "87a05578210249cb85bbe517c0de92c4"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "9cca23890d7f10c71f191d0c08b18362"
server_key_iv: "c46d7c53bbefe702f5322d53193467c0"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.502914 6666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:10.523522 6666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:10.526696 6666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:10.526916 6666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:10.527550 6666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:10.528522 6666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:10.528597 6666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.528697 6666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:10.528754 6666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.538570 6666 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:39641
I20260620 06:02:10.538591 6794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:39641 every 8 connection(s)
I20260620 06:02:10.539474 6666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:10.545058 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6666
I20260620 06:02:10.545199 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:10.545532 3532 external_mini_cluster.cc:1428] Setting key b6e009a327553aed35333726229ba948
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:10.554749 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:10.541375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37893 (local address 127.3.115.62:35215)
0620 06:02:10.541543 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:10.541548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:10.542408 (+ 860us) server_negotiation.cc:407] Connection header received
0620 06:02:10.543848 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:10.543851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:10.543942 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:10.544025 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:10.545506 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.546179 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.547456 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.547700 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.547810 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:10.550658 (+ 2848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:10.550665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:10.550671 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:10.550706 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:10.552334 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:10.553066 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:10.553069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:10.553070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:10.553151 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:10.553565 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:10.553567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:10.553568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:10.553745 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:10.553817 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:10.554506 (+ 689us) server_negotiation.cc:299] Negotiation successful
0620 06:02:10.554624 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:02:10.556051 6797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:10.541801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35215 (local address 127.3.115.1:37893)
0620 06:02:10.542257 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:10.542299 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:10.543644 (+ 1345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:10.544206 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:10.544213 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:10.544578 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:10.545351 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:10.545375 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.546327 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.546330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:10.547330 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:10.547339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.547849 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.547853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:10.548025 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:10.548711 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:10.548732 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:10.550475 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:10.552466 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:10.552475 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:10.552495 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:10.552946 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:10.553251 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:10.553254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:10.553256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:10.553462 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:10.553852 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:10.553859 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:10.554233 (+ 374us) client_negotiation.cc:769] Sending connection context
0620 06:02:10.554468 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 06:02:10.554868 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":128,"threads_started":1}
I20260620 06:02:10.557338 6795 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35215
I20260620 06:02:10.557612 6795 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:10.558171 6795 heartbeater.cc:511] Master 127.3.115.62:35215 requested a full tablet report, sending...
I20260620 06:02:10.559867 6604 ts_manager.cc:194] Registered new tserver with Master: 87a05578210249cb85bbe517c0de92c4 (127.3.115.1:39641)
I20260620 06:02:10.561230 6604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:37893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:10.573523 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35215
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:10.689081 6800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:10.689380 6800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:10.689426 6800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:10.693964 6800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:10.694041 6800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:10.694108 6800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:10.699604 6800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:10.700750 6800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:10.701768 6800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:10.711831 6808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.711823 6805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.711818 6806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:10.712203 6800 server_base.cc:1034] running on GCE node
I20260620 06:02:10.712786 6800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:10.713490 6800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:10.714689 6800 hybrid_clock.cc:648] HybridClock initialized: now 1781935330714657 us; error 54 us; skew 500 ppm
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:10.717558 6800 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:10.718396 6800 webserver.cc:460] Webserver started at http://127.3.115.2:40553/ using document root <none> and password file <none>
I20260620 06:02:10.718930 6800 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:10.718988 6800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:10.719158 6800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:10.721351 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "b3aaae1a5cfc4b26a4a17695646ae568"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "16a195a61a67839f2e8c3952f95bd4e0"
server_key_iv: "4aa2eae2c21a76eca6be937e46024467"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.721872 6800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "b3aaae1a5cfc4b26a4a17695646ae568"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "16a195a61a67839f2e8c3952f95bd4e0"
server_key_iv: "4aa2eae2c21a76eca6be937e46024467"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.725791 6800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:02:10.728353 6815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.729895 6800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:10.730012 6800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "b3aaae1a5cfc4b26a4a17695646ae568"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "16a195a61a67839f2e8c3952f95bd4e0"
server_key_iv: "4aa2eae2c21a76eca6be937e46024467"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.730116 6800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:10.765712 6800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:10.770493 6800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:10.770787 6800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:10.771540 6800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:10.772405 6800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:10.772486 6800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.772553 6800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:10.772583 6800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.781733 6800 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:43065
I20260620 06:02:10.781752 6928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:43065 every 8 connection(s)
I20260620 06:02:10.782718 6800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:10.790639 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6800
I20260620 06:02:10.790766 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:10.791113 3532 external_mini_cluster.cc:1428] Setting key 3c8bbf8c304da9b504a61378d371feca
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:10.797935 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:10.784716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:34265 (local address 127.3.115.62:35215)
0620 06:02:10.784855 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:10.784859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:10.785828 (+ 969us) server_negotiation.cc:407] Connection header received
0620 06:02:10.787176 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:10.787179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:10.787225 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:10.787314 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:10.788446 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.789039 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.790550 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.790996 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.791102 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:10.793707 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:10.793710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:10.793712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:10.793735 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:10.795473 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:10.796150 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:10.796153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:10.796155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:10.796234 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:10.796759 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:10.796762 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:10.796764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:10.796969 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:10.797093 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:10.797633 (+ 540us) server_negotiation.cc:299] Negotiation successful
0620 06:02:10.797766 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:02:10.798848 6931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:10.785151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35215 (local address 127.3.115.2:34265)
0620 06:02:10.785664 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:10.785707 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:10.786972 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:10.787466 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:10.787472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:10.787850 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:10.788322 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:10.788334 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.789169 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.789173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:10.790329 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:10.790361 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:10.791167 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:10.791174 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:10.791398 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:10.791977 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:10.791993 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:10.793533 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:10.795615 (+ 2082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:10.795624 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:10.795645 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:10.796042 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:10.796330 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:10.796336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:10.796341 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:10.796571 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:10.797093 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:10.797101 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:10.797382 (+ 281us) client_negotiation.cc:769] Sending connection context
0620 06:02:10.797592 (+ 210us) client_negotiation.cc:240] Negotiation successful
0620 06:02:10.797970 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":126,"threads_started":1}
I20260620 06:02:10.800136 6929 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35215
I20260620 06:02:10.800412 6929 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:02:10.801023 6929 heartbeater.cc:511] Master 127.3.115.62:35215 requested a full tablet report, sending...
I20260620 06:02:10.802193 6604 ts_manager.cc:194] Registered new tserver with Master: b3aaae1a5cfc4b26a4a17695646ae568 (127.3.115.2:43065)
I20260620 06:02:10.802850 6604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:34265
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:10.819473 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35215
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:10.931932 6934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:10.932236 6934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:10.932281 6934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:10.936566 6934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:10.936642 6934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:10.936760 6934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:10.942740 6934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:10.943987 6934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:10.945232 6934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:10.955396 6939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.955382 6942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:10.955382 6940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:10.955809 6934 server_base.cc:1034] running on GCE node
I20260620 06:02:10.956321 6934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:10.957086 6934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:10.958252 6934 hybrid_clock.cc:648] HybridClock initialized: now 1781935330958225 us; error 43 us; skew 500 ppm
Jun 20 06:02:10 dist-test-slave-4h14 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:10.961400 6934 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:10.962302 6934 webserver.cc:460] Webserver started at http://127.3.115.3:40307/ using document root <none> and password file <none>
I20260620 06:02:10.962865 6934 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:10.962926 6934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:10.963099 6934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:10.965251 6934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "d6774bb7fdf7404bb4673284e2e9c991"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "d32873fee3b8a8211e5c768b7fc196e7"
server_key_iv: "c722a13c5dad504519e0d9eba59266f4"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.965849 6934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "d6774bb7fdf7404bb4673284e2e9c991"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "d32873fee3b8a8211e5c768b7fc196e7"
server_key_iv: "c722a13c5dad504519e0d9eba59266f4"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.969725 6934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:02:10.972291 6949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.973738 6934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:02:10.973847 6934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "d6774bb7fdf7404bb4673284e2e9c991"
format_stamp: "Formatted at 2026-06-20 06:02:10 on dist-test-slave-4h14"
server_key: "d32873fee3b8a8211e5c768b7fc196e7"
server_key_iv: "c722a13c5dad504519e0d9eba59266f4"
server_key_version: "encryptionkey@0"
I20260620 06:02:10.973950 6934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:10.987425 6934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:10.990962 6934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:10.991128 6934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:10.991658 6934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:10.992453 6934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:10.992503 6934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:10.992543 6934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:10.992559 6934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:11.001761 6934 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:37743
I20260620 06:02:11.001825 7062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:37743 every 8 connection(s)
I20260620 06:02:11.002722 6934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:11.006246 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 6934
I20260620 06:02:11.006369 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:11.006644 3532 external_mini_cluster.cc:1428] Setting key f90259d4c992820b34765ca155ebbccd
Jun 20 06:02:11 dist-test-slave-4h14 krb5kdc[6559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935330, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:11.016708 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:11.004633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:57669 (local address 127.3.115.62:35215)
0620 06:02:11.004831 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:11.004838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:11.005615 (+ 777us) server_negotiation.cc:407] Connection header received
0620 06:02:11.007376 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:11.007379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:11.007436 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:11.007522 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:11.009097 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:11.009632 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:11.010741 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:11.010950 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:11.011030 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:11.013443 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:11.013446 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:11.013448 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:11.013470 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:11.014784 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:11.015255 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:11.015257 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:11.015259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:11.015330 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:11.015677 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:11.015679 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:11.015680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:11.015843 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:11.015912 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:11.016437 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 06:02:11.016530 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260620 06:02:11.017680 7065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:11.004996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35215 (local address 127.3.115.3:57669)
0620 06:02:11.005439 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:11.005481 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:11.007133 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:11.007746 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:11.007755 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:11.008283 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:11.008943 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:11.008966 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:11.009734 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:11.009737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:11.010637 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:11.010645 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:11.011077 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:11.011080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:11.011230 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:11.011784 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:11.011800 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:11.013292 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:11.014883 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:11.014888 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:11.014900 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:11.015162 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:11.015435 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:11.015438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:11.015440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:11.015584 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:11.015958 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:11.015963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:11.016206 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 06:02:11.016366 (+ 160us) client_negotiation.cc:240] Negotiation successful
0620 06:02:11.016716 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":108,"threads_started":1}
I20260620 06:02:11.018930 7063 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35215
I20260620 06:02:11.019184 7063 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:11.019826 7063 heartbeater.cc:511] Master 127.3.115.62:35215 requested a full tablet report, sending...
I20260620 06:02:11.020941 6604 ts_manager.cc:194] Registered new tserver with Master: d6774bb7fdf7404bb4673284e2e9c991 (127.3.115.3:37743)
I20260620 06:02:11.021500 6604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:57669
I20260620 06:02:11.032630 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:02:11 dist-test-slave-4h14 krb5kdc[6559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935331, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 06:02:11 dist-test-slave-4h14 krb5kdc[6559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935331, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:11.057778 6643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:11.049422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51894 (local address 127.3.115.62:35215)
0620 06:02:11.049624 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:11.049628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:11.049689 (+ 61us) server_negotiation.cc:407] Connection header received
0620 06:02:11.049805 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:11.049808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:11.049853 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:11.049924 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:11.050623 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:11.051137 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:11.051995 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:11.052215 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:11.052296 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:11.054798 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:11.054801 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:11.054803 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:11.054826 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:11.056267 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:11.056626 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:11.056628 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:11.056629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:11.056757 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:11.056985 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:11.056987 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:11.056988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:11.057220 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:11.057321 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:11.057573 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 06:02:11.057659 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260620 06:02:11.058183 6604 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51894
I20260620 06:02:11.059152 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6666
I20260620 06:02:11.064870 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6800
I20260620 06:02:11.070197 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6934
I20260620 06:02:11.075363 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 6572
2026-06-20T06:02:11Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2176 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:11 dist-test-slave-4h14 krb5kdc[7075](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:11 dist-test-slave-4h14 krb5kdc[7075](info): set up 2 sockets
Jun 20 06:02:11 dist-test-slave-4h14 krb5kdc[7075](info): commencing operation
krb5kdc: starting...
W20260620 06:02:12.121419 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:12.248468 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42799
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42799
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:12.362427 7088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:12.362797 7088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:12.362883 7088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:12.367113 7088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:12.367228 7088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:12.367269 7088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:12.367307 7088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:12.367342 7088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:12.372864 7088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:12.374107 7088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:12.375197 7088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:12.384210 7096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:12.384222 7093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:12.384207 7094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:12.384686 7088 server_base.cc:1034] running on GCE node
I20260620 06:02:12.385277 7088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:12.386430 7088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:12.387626 7088 hybrid_clock.cc:648] HybridClock initialized: now 1781935332387595 us; error 50 us; skew 500 ppm
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:12.390857 7088 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:12.391903 7088 webserver.cc:460] Webserver started at http://127.3.115.62:34197/ using document root <none> and password file <none>
I20260620 06:02:12.392534 7088 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:12.392633 7088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:12.392910 7088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:12.395099 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "d4a370e101174c46a6ed4ea7d32d7350"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "7ebadc81530c5c1d0354f99222505280"
server_key_iv: "381df921b9ace03c836a83576b21c3a5"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.395705 7088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "d4a370e101174c46a6ed4ea7d32d7350"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "7ebadc81530c5c1d0354f99222505280"
server_key_iv: "381df921b9ace03c836a83576b21c3a5"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.400257 7088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:12.403344 7103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.404888 7088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:12.405061 7088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "d4a370e101174c46a6ed4ea7d32d7350"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "7ebadc81530c5c1d0354f99222505280"
server_key_iv: "381df921b9ace03c836a83576b21c3a5"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.405375 7088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:12.425267 7088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:12.428898 7088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:12.429112 7088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:12.436496 7088 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42799
I20260620 06:02:12.436518 7155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42799 every 8 connection(s)
I20260620 06:02:12.437626 7088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:12.440537 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:12.445428 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7088
I20260620 06:02:12.445572 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:12.445876 3532 external_mini_cluster.cc:1428] Setting key 5490f6ab79267637297ed3b8087a78aa
I20260620 06:02:12.446614 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350: Bootstrap starting.
I20260620 06:02:12.449146 7156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:12.449914 7156 log.cc:824] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:12.451845 7156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350: No bootstrap required, opened a new log
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:12.455740 7156 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a370e101174c46a6ed4ea7d32d7350" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42799 } }
I20260620 06:02:12.455978 7156 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:12.456084 7156 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4a370e101174c46a6ed4ea7d32d7350, State: Initialized, Role: FOLLOWER
I20260620 06:02:12.456559 7156 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a370e101174c46a6ed4ea7d32d7350" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42799 } }
I20260620 06:02:12.456764 7156 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:12.456844 7156 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:12.456985 7156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:12.457903 7156 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a370e101174c46a6ed4ea7d32d7350" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42799 } }
I20260620 06:02:12.458316 7156 leader_election.cc:302] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4a370e101174c46a6ed4ea7d32d7350; no voters:
I20260620 06:02:12.458730 7156 leader_election.cc:288] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:12.458848 7161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:12.459069 7161 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [term 1 LEADER]: Becoming Leader. State: Replica: d4a370e101174c46a6ed4ea7d32d7350, State: Running, Role: LEADER
I20260620 06:02:12.459414 7161 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a370e101174c46a6ed4ea7d32d7350" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42799 } }
I20260620 06:02:12.460144 7156 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:12.461376 7163 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4a370e101174c46a6ed4ea7d32d7350. Latest consensus state: current_term: 1 leader_uuid: "d4a370e101174c46a6ed4ea7d32d7350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a370e101174c46a6ed4ea7d32d7350" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42799 } } }
I20260620 06:02:12.461376 7162 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4a370e101174c46a6ed4ea7d32d7350" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4a370e101174c46a6ed4ea7d32d7350" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42799 } } }
I20260620 06:02:12.461525 7162 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:12.461525 7163 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:12.461645 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:12.447377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49512 (local address 127.3.115.62:42799)
0620 06:02:12.447841 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:12.447856 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:12.447892 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:02:12.448572 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:12.448589 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:12.449019 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:12.449365 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:12.450248 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.451227 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.452143 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.452614 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.452879 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:12.455673 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:12.455679 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:12.455694 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:12.455722 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:12.457691 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:12.458169 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:12.458176 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:12.458183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:12.458333 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:12.458665 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:12.458672 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:12.458676 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:12.459217 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:12.460001 (+ 784us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:12.460159 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 06:02:12.460479 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":150,"threads_started":1}
W20260620 06:02:12.465175 7176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:02:12.465265 7176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:02:12.465320 7171 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:12.466164 7171 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:12.471963 7171 catalog_manager.cc:1347] Generated new cluster ID: 1103b24f69944e57af268fa43213abc8
I20260620 06:02:12.472059 7171 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:12.489187 7171 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:12.490535 7171 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:12.499384 7171 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4a370e101174c46a6ed4ea7d32d7350: Generated new TSK 0
I20260620 06:02:12.500219 7171 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:12.536346 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42799
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:12.687146 7182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:12.687405 7182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:12.687448 7182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:12.691962 7182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:12.692042 7182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:12.692117 7182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:12.698272 7182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:12.699371 7182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:12.700423 7182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:12.711333 7187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:12.711362 7190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:12.711414 7182 server_base.cc:1034] running on GCE node
W20260620 06:02:12.711361 7188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:12.712028 7182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:12.712615 7182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:12.713825 7182 hybrid_clock.cc:648] HybridClock initialized: now 1781935332713809 us; error 28 us; skew 500 ppm
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:12.716717 7182 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:12.717569 7182 webserver.cc:460] Webserver started at http://127.3.115.1:41425/ using document root <none> and password file <none>
I20260620 06:02:12.718128 7182 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:12.718190 7182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:12.718405 7182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:12.720454 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "c32c002062d64e44aced51f9dc53d5eb"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "5d66b8a41279c5ed06c83930737a8e58"
server_key_iv: "02c105f417820781a4bd46fcbe546835"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.721086 7182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "c32c002062d64e44aced51f9dc53d5eb"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "5d66b8a41279c5ed06c83930737a8e58"
server_key_iv: "02c105f417820781a4bd46fcbe546835"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.724946 7182 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:02:12.727448 7197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.728739 7182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:12.728888 7182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "c32c002062d64e44aced51f9dc53d5eb"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "5d66b8a41279c5ed06c83930737a8e58"
server_key_iv: "02c105f417820781a4bd46fcbe546835"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.729010 7182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:12.754580 7182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:12.757869 7182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:12.758090 7182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:12.758730 7182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:12.759596 7182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:12.759665 7182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.759720 7182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:12.759755 7182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.769229 7182 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:38915
I20260620 06:02:12.769253 7310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:38915 every 8 connection(s)
I20260620 06:02:12.770190 7182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:12.774497 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7182
I20260620 06:02:12.774639 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:12.774912 3532 external_mini_cluster.cc:1428] Setting key 774c928e3853efc72ce2131a5950a472
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:02:12.787943 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:12.771987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37381 (local address 127.3.115.62:42799)
0620 06:02:12.772166 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:12.772170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:12.773042 (+ 872us) server_negotiation.cc:407] Connection header received
0620 06:02:12.774423 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:12.774427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:12.774490 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:12.774594 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:12.775925 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.776714 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.778609 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.778998 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.779115 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:12.783243 (+ 4128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:12.783249 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:12.783254 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:12.783292 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:12.785254 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:12.785963 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:12.785966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:12.785968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:12.786128 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:12.786524 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:12.786527 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:12.786529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:12.786785 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:12.786903 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:12.787637 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 06:02:12.787776 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 06:02:12.788966 7313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:12.772395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42799 (local address 127.3.115.1:37381)
0620 06:02:12.772891 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:12.772935 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:12.774215 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:12.774804 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:12.774811 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:12.775229 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:12.775796 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:12.775810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.776868 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.776874 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:12.778488 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:12.778500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.779300 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.779304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:12.779524 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:12.780524 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:12.780549 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:12.783002 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:12.785416 (+ 2414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:12.785425 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:12.785447 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:12.785840 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:12.786244 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:12.786249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:12.786251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:12.786410 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:12.786899 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:12.786906 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:12.787293 (+ 387us) client_negotiation.cc:769] Sending connection context
0620 06:02:12.787592 (+ 299us) client_negotiation.cc:240] Negotiation successful
0620 06:02:12.787955 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":120,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:12.790342 7311 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42799
I20260620 06:02:12.790649 7311 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:12.791165 7311 heartbeater.cc:511] Master 127.3.115.62:42799 requested a full tablet report, sending...
I20260620 06:02:12.792917 7120 ts_manager.cc:194] Registered new tserver with Master: c32c002062d64e44aced51f9dc53d5eb (127.3.115.1:38915)
I20260620 06:02:12.794615 7120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:37381
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:12.805174 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42799
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:12.915403 7316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:12.915678 7316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:12.915721 7316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:12.920092 7316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:12.920159 7316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:12.920218 7316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:12.925653 7316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:12.926700 7316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:12.927713 7316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:12.937659 7321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:12.937616 7316 server_base.cc:1034] running on GCE node
W20260620 06:02:12.937598 7322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:12.937631 7324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:12.938259 7316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:12.938832 7316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:12.940017 7316 hybrid_clock.cc:648] HybridClock initialized: now 1781935332939982 us; error 51 us; skew 500 ppm
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:12.942734 7316 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:12.943550 7316 webserver.cc:460] Webserver started at http://127.3.115.2:33659/ using document root <none> and password file <none>
I20260620 06:02:12.944113 7316 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:12.944173 7316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:12.944339 7316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:12.946360 7316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "d10bd6a0dfba4cce964a1bfda5d1f25a"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "a80c1586e68044a5c6259d42dc0c28c7"
server_key_iv: "d6c08f038e2986dad358f4488f4d98e6"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.946869 7316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "d10bd6a0dfba4cce964a1bfda5d1f25a"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "a80c1586e68044a5c6259d42dc0c28c7"
server_key_iv: "d6c08f038e2986dad358f4488f4d98e6"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.950634 7316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:12.953207 7331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.954492 7316 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:12.954599 7316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "d10bd6a0dfba4cce964a1bfda5d1f25a"
format_stamp: "Formatted at 2026-06-20 06:02:12 on dist-test-slave-4h14"
server_key: "a80c1586e68044a5c6259d42dc0c28c7"
server_key_iv: "d6c08f038e2986dad358f4488f4d98e6"
server_key_version: "encryptionkey@0"
I20260620 06:02:12.954690 7316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:12.969866 7316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:12.973078 7316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:12.973240 7316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:12.973774 7316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:12.974565 7316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:12.974611 7316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.974653 7316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:12.974668 7316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:12.983209 7316 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:32955
I20260620 06:02:12.983227 7444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:32955 every 8 connection(s)
I20260620 06:02:12.984109 7316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:12.991665 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7316
I20260620 06:02:12.991802 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:12.992072 3532 external_mini_cluster.cc:1428] Setting key 82263facccaa6e8fec0fb768f62602ed
Jun 20 06:02:12 dist-test-slave-4h14 krb5kdc[7075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935332, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:12.998728 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:12.985841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:34759 (local address 127.3.115.62:42799)
0620 06:02:12.985969 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:12.985973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:12.986819 (+ 846us) server_negotiation.cc:407] Connection header received
0620 06:02:12.988188 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:12.988192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:12.988245 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:12.988352 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:12.989565 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.990101 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.991243 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.991549 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.991668 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:12.994648 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:12.994654 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:12.994661 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:12.994694 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:12.996313 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:12.996941 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:12.996947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:12.996953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:12.997048 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:12.997435 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:12.997441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:12.997446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:12.997709 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:12.997825 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:12.998339 (+ 514us) server_negotiation.cc:299] Negotiation successful
0620 06:02:12.998484 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:02:12.999571 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:12.986231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42799 (local address 127.3.115.2:34759)
0620 06:02:12.986673 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:12.986715 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:12.987982 (+ 1267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:12.988523 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:12.988530 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:12.988954 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:12.989422 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:12.989435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.990227 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.990233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:12.991119 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:12.991128 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:12.991833 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:12.991844 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:12.992192 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:12.992923 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:12.992939 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:12.994475 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:12.996441 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:12.996446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:12.996459 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:12.996840 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:12.997176 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:12.997179 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:12.997180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:12.997317 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:12.997831 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:12.997836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:12.998073 (+ 237us) client_negotiation.cc:769] Sending connection context
0620 06:02:12.998230 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 06:02:12.998487 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
I20260620 06:02:13.000859 7445 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42799
I20260620 06:02:13.001135 7445 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:13.001762 7445 heartbeater.cc:511] Master 127.3.115.62:42799 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:02:13.002894 7120 ts_manager.cc:194] Registered new tserver with Master: d10bd6a0dfba4cce964a1bfda5d1f25a (127.3.115.2:32955)
I20260620 06:02:13.003583 7120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:34759
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:13.020118 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42799
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:13.131270 7450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:13.131623 7450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:13.131703 7450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:13.136296 7450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:13.136382 7450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:13.136471 7450 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:13.141902 7450 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42799
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:13.143010 7450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:13.144119 7450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:13.153720 7455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:13.153728 7456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:13.153728 7458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:13.154045 7450 server_base.cc:1034] running on GCE node
I20260620 06:02:13.154546 7450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:13.155134 7450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:13.156322 7450 hybrid_clock.cc:648] HybridClock initialized: now 1781935333156286 us; error 48 us; skew 500 ppm
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935333, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:13.159288 7450 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:13.160226 7450 webserver.cc:460] Webserver started at http://127.3.115.3:44319/ using document root <none> and password file <none>
I20260620 06:02:13.160872 7450 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:13.160965 7450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:13.161208 7450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:13.163244 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "692630d5a6614e4db8db393c8864c86d"
format_stamp: "Formatted at 2026-06-20 06:02:13 on dist-test-slave-4h14"
server_key: "70ca8548f8dee3634ceebf58018f0cff"
server_key_iv: "a4d0ff9885dd118bca4aac849e8d0cc1"
server_key_version: "encryptionkey@0"
I20260620 06:02:13.163760 7450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "692630d5a6614e4db8db393c8864c86d"
format_stamp: "Formatted at 2026-06-20 06:02:13 on dist-test-slave-4h14"
server_key: "70ca8548f8dee3634ceebf58018f0cff"
server_key_iv: "a4d0ff9885dd118bca4aac849e8d0cc1"
server_key_version: "encryptionkey@0"
I20260620 06:02:13.167713 7450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:02:13.170222 7465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:13.171489 7450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:13.171634 7450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "692630d5a6614e4db8db393c8864c86d"
format_stamp: "Formatted at 2026-06-20 06:02:13 on dist-test-slave-4h14"
server_key: "70ca8548f8dee3634ceebf58018f0cff"
server_key_iv: "a4d0ff9885dd118bca4aac849e8d0cc1"
server_key_version: "encryptionkey@0"
I20260620 06:02:13.171739 7450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:13.192204 7450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:13.195292 7450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:13.195511 7450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:13.196135 7450 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:13.197083 7450 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:13.197156 7450 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:13.197235 7450 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:13.197278 7450 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:13.206215 7450 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:43483
I20260620 06:02:13.206238 7578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:43483 every 8 connection(s)
I20260620 06:02:13.207108 7450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:13.216842 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7450
I20260620 06:02:13.216982 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935333, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:13.217346 3532 external_mini_cluster.cc:1428] Setting key 5ae0af62d2f4c94966c495722ba526d5
I20260620 06:02:13.221774 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:13.208907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:50655 (local address 127.3.115.62:42799)
0620 06:02:13.209064 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:13.209069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:13.210142 (+ 1073us) server_negotiation.cc:407] Connection header received
0620 06:02:13.211579 (+ 1437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:13.211582 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:13.211630 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:13.211709 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:13.212972 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.213496 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.214668 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.214882 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.214971 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:13.217899 (+ 2928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:13.217903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:13.217905 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:13.217933 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:13.219659 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:13.220156 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:13.220159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:13.220160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:13.220238 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:13.220700 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:13.220706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:13.220711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:13.220947 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:13.221042 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:13.221547 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 06:02:13.221655 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:02:13.222674 7581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:13.209456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42799 (local address 127.3.115.3:50655)
0620 06:02:13.209948 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:13.209993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:13.211362 (+ 1369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:13.211876 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:13.211883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:13.212282 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:13.212842 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:13.212858 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.213605 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.213607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:13.214536 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:13.214545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.215017 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.215019 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:13.215164 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:13.215843 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:13.215860 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:13.217682 (+ 1822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:13.219775 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:13.219779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:13.219794 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:13.220074 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:13.220346 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:13.220349 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:13.220351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:13.220517 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:13.221062 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:13.221067 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:13.221323 (+ 256us) client_negotiation.cc:769] Sending connection context
0620 06:02:13.221496 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 06:02:13.221780 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":108,"threads_started":1}
I20260620 06:02:13.223937 7579 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42799
I20260620 06:02:13.224202 7579 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:13.224807 7579 heartbeater.cc:511] Master 127.3.115.62:42799 requested a full tablet report, sending...
I20260620 06:02:13.225883 7120 ts_manager.cc:194] Registered new tserver with Master: 692630d5a6614e4db8db393c8864c86d (127.3.115.3:43483)
I20260620 06:02:13.226459 7120 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:50655
I20260620 06:02:13.231299 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7075](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:02:13.257364 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:13.247561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55026 (local address 127.3.115.1:38915)
0620 06:02:13.247895 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:13.247912 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:13.247945 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:02:13.248014 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:13.248020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:13.248219 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:13.248377 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:13.249178 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.249954 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.250819 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.251202 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.251324 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:13.253853 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:13.253862 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:13.253876 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:13.253901 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:13.255472 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:13.255927 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:13.255933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:13.255934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:13.256013 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:13.256300 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:13.256305 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:13.256306 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:13.256557 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:13.256808 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:13.256999 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 06:02:13.257163 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":118,"threads_started":1}
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7075](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:13.268877 7159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:13.260803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49518 (local address 127.3.115.62:42799)
0620 06:02:13.260968 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:13.260973 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:13.261098 (+ 125us) server_negotiation.cc:407] Connection header received
0620 06:02:13.261268 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:13.261272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:13.261316 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:13.261360 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:13.262001 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.262603 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.263372 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:13.263582 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:13.263655 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:13.265997 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:13.266000 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:13.266002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:13.266026 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:13.267419 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:13.267778 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:13.267780 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:13.267781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:13.267851 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:13.268066 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:13.268069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:13.268070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:13.268270 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:13.268368 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:13.268538 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:02:13.268635 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 06:02:13.269295 7120 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49518
I20260620 06:02:13.269904 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7182
I20260620 06:02:13.275439 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7316
I20260620 06:02:13.280966 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7450
I20260620 06:02:13.287832 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7088
2026-06-20T06:02:13Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2213 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7591](info): set up 2 sockets
Jun 20 06:02:13 dist-test-slave-4h14 krb5kdc[7591](info): commencing operation
krb5kdc: starting...
W20260620 06:02:14.325399 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:14 dist-test-slave-4h14 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:14.451481 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34871
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:34871
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:14.566067 7604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:14.566402 7604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:14.566478 7604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:14.570608 7604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:14.570710 7604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:14.570749 7604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:14.570787 7604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:14.570819 7604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:14.576143 7604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:34871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7604
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:14.577275 7604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:14.578858 7604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:14.587294 7612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:14.587284 7609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:14.587402 7604 server_base.cc:1034] running on GCE node
W20260620 06:02:14.587312 7610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:14.588243 7604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:14.589493 7604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:14.590695 7604 hybrid_clock.cc:648] HybridClock initialized: now 1781935334590668 us; error 43 us; skew 500 ppm
Jun 20 06:02:14 dist-test-slave-4h14 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:14.598354 7604 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:14.599210 7604 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:14.599315 7604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:14.599575 7604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:14.601868 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "00f16ef762944128b8fdf852735f16ba"
format_stamp: "Formatted at 2026-06-20 06:02:14 on dist-test-slave-4h14"
server_key: "acb5a2019b0c5879d2f0641125e5f177"
server_key_iv: "2d63acb3e57d531326e875e20e3a038d"
server_key_version: "encryptionkey@0"
I20260620 06:02:14.602511 7604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "00f16ef762944128b8fdf852735f16ba"
format_stamp: "Formatted at 2026-06-20 06:02:14 on dist-test-slave-4h14"
server_key: "acb5a2019b0c5879d2f0641125e5f177"
server_key_iv: "2d63acb3e57d531326e875e20e3a038d"
server_key_version: "encryptionkey@0"
I20260620 06:02:14.606338 7604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:14.608886 7618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:14.610180 7604 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:14.610332 7604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "00f16ef762944128b8fdf852735f16ba"
format_stamp: "Formatted at 2026-06-20 06:02:14 on dist-test-slave-4h14"
server_key: "acb5a2019b0c5879d2f0641125e5f177"
server_key_iv: "2d63acb3e57d531326e875e20e3a038d"
server_key_version: "encryptionkey@0"
I20260620 06:02:14.610452 7604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:14.631067 7604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:14.634534 7604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:14.634704 7604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:14.641848 7604 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:34871
I20260620 06:02:14.642125 7670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:34871 every 8 connection(s)
I20260620 06:02:14.642589 7604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:14.645604 7671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:14.648300 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7604
I20260620 06:02:14.648478 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:14.648866 3532 external_mini_cluster.cc:1428] Setting key 869f882bb1267253f8da4e3b0fcfdb5d
I20260620 06:02:14.652011 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba: Bootstrap starting.
I20260620 06:02:14.654408 7671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:14.655203 7671 log.cc:824] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:14.657251 7671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba: No bootstrap required, opened a new log
Jun 20 06:02:14 dist-test-slave-4h14 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:14.661444 7671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f16ef762944128b8fdf852735f16ba" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34871 } }
I20260620 06:02:14.661731 7671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:14.661813 7671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f16ef762944128b8fdf852735f16ba, State: Initialized, Role: FOLLOWER
I20260620 06:02:14.662309 7671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f16ef762944128b8fdf852735f16ba" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34871 } }
I20260620 06:02:14.662472 7671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:14.662560 7671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:14.662701 7671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:14.663534 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:14.650315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.3.115.62:34871)
0620 06:02:14.650787 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:14.650801 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:14.650838 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:02:14.651457 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:14.651476 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:14.651834 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:14.652194 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:14.653016 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:14.654018 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:14.654940 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:14.655487 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:14.655669 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:14.658453 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:14.658458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:14.658473 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:14.658500 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:14.660317 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:14.660869 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:14.660874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:14.660879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:14.661002 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:14.661318 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:14.661321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:14.661322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:14.661814 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:14.662026 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:14.662317 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 06:02:14.662699 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":140,"threads_started":1}
I20260620 06:02:14.663720 7671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f16ef762944128b8fdf852735f16ba" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34871 } }
I20260620 06:02:14.664196 7671 leader_election.cc:302] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f16ef762944128b8fdf852735f16ba; no voters:
I20260620 06:02:14.664541 7671 leader_election.cc:288] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:14.664697 7676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:14.664956 7676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [term 1 LEADER]: Becoming Leader. State: Replica: 00f16ef762944128b8fdf852735f16ba, State: Running, Role: LEADER
I20260620 06:02:14.665336 7676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f16ef762944128b8fdf852735f16ba" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34871 } }
I20260620 06:02:14.665956 7671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:14.666752 7678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00f16ef762944128b8fdf852735f16ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f16ef762944128b8fdf852735f16ba" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34871 } } }
I20260620 06:02:14.666755 7676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f16ef762944128b8fdf852735f16ba. Latest consensus state: current_term: 1 leader_uuid: "00f16ef762944128b8fdf852735f16ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f16ef762944128b8fdf852735f16ba" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34871 } } }
I20260620 06:02:14.667050 7678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:14.667352 7676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:14.667557 7685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:14.671226 7685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:14.677672 7685 catalog_manager.cc:1347] Generated new cluster ID: 6c957d000de44c62b5e75034202635f1
I20260620 06:02:14.677750 7685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:14.691401 7685 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:14.692720 7685 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:14.699167 7685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba: Generated new TSK 0
I20260620 06:02:14.699987 7685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:14.731467 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34871
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:14.846410 7697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:14.846735 7697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:14.846817 7697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:14.851055 7697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:14.851162 7697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:14.851262 7697 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:14.856853 7697 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:14.857980 7697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:14.859042 7697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:14.869074 7705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:14.869235 7697 server_base.cc:1034] running on GCE node
W20260620 06:02:14.869170 7702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:14.869074 7703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:14.869796 7697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:14.870481 7697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:14.871665 7697 hybrid_clock.cc:648] HybridClock initialized: now 1781935334871632 us; error 46 us; skew 500 ppm
Jun 20 06:02:14 dist-test-slave-4h14 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:14.874554 7697 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:14.875310 7697 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:14.875404 7697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:14.875641 7697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:14.877847 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "817b1feb67bf4d9fb2696ab7fb9004e1"
format_stamp: "Formatted at 2026-06-20 06:02:14 on dist-test-slave-4h14"
server_key: "827a2f6fb1a6622db8e7a0b193ca6072"
server_key_iv: "33f63294b65dc71870eb6c15004b3658"
server_key_version: "encryptionkey@0"
I20260620 06:02:14.878441 7697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "817b1feb67bf4d9fb2696ab7fb9004e1"
format_stamp: "Formatted at 2026-06-20 06:02:14 on dist-test-slave-4h14"
server_key: "827a2f6fb1a6622db8e7a0b193ca6072"
server_key_iv: "33f63294b65dc71870eb6c15004b3658"
server_key_version: "encryptionkey@0"
I20260620 06:02:14.882179 7697 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:02:14.884599 7711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:14.885919 7697 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:14.886042 7697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "817b1feb67bf4d9fb2696ab7fb9004e1"
format_stamp: "Formatted at 2026-06-20 06:02:14 on dist-test-slave-4h14"
server_key: "827a2f6fb1a6622db8e7a0b193ca6072"
server_key_iv: "33f63294b65dc71870eb6c15004b3658"
server_key_version: "encryptionkey@0"
I20260620 06:02:14.886188 7697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:14.901573 7697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:14.905103 7697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:14.905265 7697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:14.905808 7697 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:14.906618 7697 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:14.906665 7697 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:14.906704 7697 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:14.906720 7697 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:14.916173 7697 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35849
I20260620 06:02:14.916198 7824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35849 every 8 connection(s)
I20260620 06:02:14.916827 7697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:14.918282 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7697
I20260620 06:02:14.918411 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:14.918733 3532 external_mini_cluster.cc:1428] Setting key a85005459b8c480792cd8a9bb9e04a58
Jun 20 06:02:14 dist-test-slave-4h14 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935334, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:14.932878 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:14.918720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:49109 (local address 127.3.115.62:34871)
0620 06:02:14.918841 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:14.918845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:14.919905 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 06:02:14.921846 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:14.921849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:14.921915 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:14.922016 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:14.923249 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:14.924175 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:14.925481 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:14.925934 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:14.926061 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:14.928930 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:14.928934 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:14.928936 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:14.928960 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:14.930387 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:14.930961 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:14.930966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:14.930970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:14.931061 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:14.931542 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:14.931566 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:14.931570 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:14.931849 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:14.931961 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:14.932503 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 06:02:14.932684 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:02:14.933976 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:14.919223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34871 (local address 127.3.115.1:49109)
0620 06:02:14.919738 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:14.919792 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:14.921607 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:14.922227 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:14.922234 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:14.922593 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:14.923098 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:14.923111 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:14.924303 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:14.924305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:14.925356 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:14.925365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:14.926110 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:14.926113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:14.926268 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:14.926908 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:14.926924 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:14.928761 (+ 1837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:14.930512 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:14.930519 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:14.930535 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:14.930811 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:14.931218 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:14.931224 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:14.931229 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:14.931426 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:14.931956 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:14.931961 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:14.932243 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 06:02:14.932401 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 06:02:14.932792 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":134,"threads_started":1}
I20260620 06:02:14.935338 7825 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34871
I20260620 06:02:14.935616 7825 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:14.936090 7825 heartbeater.cc:511] Master 127.3.115.62:34871 requested a full tablet report, sending...
I20260620 06:02:14.937680 7635 ts_manager.cc:194] Registered new tserver with Master: 817b1feb67bf4d9fb2696ab7fb9004e1 (127.3.115.1:35849)
I20260620 06:02:14.938813 7635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:49109
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:14.948002 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34871
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:15.059247 7830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:15.059545 7830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:15.059626 7830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:15.063758 7830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:15.063850 7830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:15.063957 7830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:15.069566 7830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:15.070688 7830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:15.071755 7830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:15.081234 7838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:15.081182 7836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:15.081434 7830 server_base.cc:1034] running on GCE node
W20260620 06:02:15.081182 7835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:15.081876 7830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:15.082548 7830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:15.083745 7830 hybrid_clock.cc:648] HybridClock initialized: now 1781935335083729 us; error 33 us; skew 500 ppm
Jun 20 06:02:15 dist-test-slave-4h14 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:15.086504 7830 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:15.087213 7830 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:15.087276 7830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:15.087440 7830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:15.089571 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "25009ccbc95847b3a9af7069627e6757"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "01981c4e26c2c9e51f74daf810673f5f"
server_key_iv: "566379e0f9f1fc2daefe614399269a18"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.090168 7830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "25009ccbc95847b3a9af7069627e6757"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "01981c4e26c2c9e51f74daf810673f5f"
server_key_iv: "566379e0f9f1fc2daefe614399269a18"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.093945 7830 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:15.096321 7844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.097637 7830 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:15.097733 7830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "25009ccbc95847b3a9af7069627e6757"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "01981c4e26c2c9e51f74daf810673f5f"
server_key_iv: "566379e0f9f1fc2daefe614399269a18"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.097816 7830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:15.123306 7830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:15.126309 7830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:15.126466 7830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:15.127007 7830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:15.127785 7830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:15.127837 7830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.127877 7830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:15.127900 7830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.136533 7830 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:39265
I20260620 06:02:15.136554 7957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:39265 every 8 connection(s)
I20260620 06:02:15.137379 7830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:15.144579 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7830
I20260620 06:02:15.144761 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:15.145056 3532 external_mini_cluster.cc:1428] Setting key 2bb236640ce8e3cf355ef0d23a4d1575
Jun 20 06:02:15 dist-test-slave-4h14 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:15.152004 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.139030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:42089 (local address 127.3.115.62:34871)
0620 06:02:15.139213 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.139217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.140008 (+ 791us) server_negotiation.cc:407] Connection header received
0620 06:02:15.141384 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.141388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.141447 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.141560 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:15.142705 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.143234 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.144401 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.144653 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.144774 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.147906 (+ 3132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:15.147912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:15.147920 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:15.147943 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:15.149654 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:15.150309 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:15.150312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:15.150314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:15.150395 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:15.150802 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:15.150804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:15.150805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:15.150984 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:15.151061 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.151770 (+ 709us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.151881 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260620 06:02:15.153438 7960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.139378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34871 (local address 127.3.115.2:42089)
0620 06:02:15.139849 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.139911 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.141176 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.141724 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.141732 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.142091 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:15.142560 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.142573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.143367 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.143374 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.144258 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.144266 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.144816 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.144821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.144994 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.145852 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:15.145871 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:15.147733 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:15.149776 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:15.149785 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:15.149806 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:15.150198 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:15.150496 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:15.150502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:15.150507 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:15.150705 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:15.151096 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:15.151105 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:15.151482 (+ 377us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.151722 (+ 240us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.152148 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":124,"threads_started":1}
I20260620 06:02:15.154717 7958 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34871
I20260620 06:02:15.155028 7958 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:15.155619 7958 heartbeater.cc:511] Master 127.3.115.62:34871 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:02:15.156883 7635 ts_manager.cc:194] Registered new tserver with Master: 25009ccbc95847b3a9af7069627e6757 (127.3.115.2:39265)
I20260620 06:02:15.157480 7635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:42089
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:15.173812 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34871
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:15.287830 7963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:15.288141 7963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:15.288223 7963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:15.292527 7963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:15.292615 7963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:15.292732 7963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:15.298097 7963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34871
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:15.299183 7963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:15.300220 7963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:15.309998 7969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:15.309998 7971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:15.310035 7968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:15.310398 7963 server_base.cc:1034] running on GCE node
I20260620 06:02:15.310817 7963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:15.311439 7963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:15.312603 7963 hybrid_clock.cc:648] HybridClock initialized: now 1781935335312587 us; error 31 us; skew 500 ppm
Jun 20 06:02:15 dist-test-slave-4h14 krb5kdc[7591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:15.315492 7963 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:15.316220 7963 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:15.316310 7963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:15.316540 7963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:15.318632 7963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "158c8b25f1ee4421858fcb3dd76f3870"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "0a3cef07b8c3e33016b53da6fa7cd1bd"
server_key_iv: "881d237377c9af3e10ba46287e2c5c50"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.319218 7963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "158c8b25f1ee4421858fcb3dd76f3870"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "0a3cef07b8c3e33016b53da6fa7cd1bd"
server_key_iv: "881d237377c9af3e10ba46287e2c5c50"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.322896 7963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260620 06:02:15.325393 7977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.326624 7963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:02:15.326757 7963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "158c8b25f1ee4421858fcb3dd76f3870"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "0a3cef07b8c3e33016b53da6fa7cd1bd"
server_key_iv: "881d237377c9af3e10ba46287e2c5c50"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.326869 7963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:15.352192 7963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:15.355895 7963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:15.356122 7963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:15.356822 7963 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:15.357714 7963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:15.357764 7963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.357805 7963 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:15.357848 7963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.367552 7963 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40367
I20260620 06:02:15.367540 8090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40367 every 8 connection(s)
I20260620 06:02:15.368302 7963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:15.369856 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 7963
I20260620 06:02:15.369984 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:15.370323 3532 external_mini_cluster.cc:1428] Setting key 2016c52d92e9c91a3c9f178cd056fb97
Jun 20 06:02:15 dist-test-slave-4h14 krb5kdc[7591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935335, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:15.383529 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.370429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:39867 (local address 127.3.115.62:34871)
0620 06:02:15.370581 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.370586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.372289 (+ 1703us) server_negotiation.cc:407] Connection header received
0620 06:02:15.373745 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.373748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.373800 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.373884 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:15.375119 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.375649 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.376876 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.377107 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.377222 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.379711 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:15.379722 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:15.379724 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:15.379747 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:15.381377 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:15.381864 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:15.381866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:15.381868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:15.381942 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:15.382295 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:15.382298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:15.382299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:15.382475 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:15.382552 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.383194 (+ 642us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.383310 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:02:15.384609 8093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.370992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34871 (local address 127.3.115.3:39867)
0620 06:02:15.372136 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.372181 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.373527 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.374094 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.374101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.374477 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:15.374972 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.374986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.375780 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.375783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.376749 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.376758 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.377260 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.377265 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.377441 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.378072 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:15.378088 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:15.379545 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:15.381483 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:15.381488 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:15.381502 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:15.381771 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:15.382048 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:15.382050 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:15.382052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:15.382208 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:15.382590 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:15.382598 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:15.382912 (+ 314us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.383099 (+ 187us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.383480 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":146,"threads_started":1}
I20260620 06:02:15.385692 8091 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34871
I20260620 06:02:15.385946 8091 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:15.386404 8091 heartbeater.cc:511] Master 127.3.115.62:34871 requested a full tablet report, sending...
I20260620 06:02:15.387398 7635 ts_manager.cc:194] Registered new tserver with Master: 158c8b25f1ee4421858fcb3dd76f3870 (127.3.115.3:40367)
I20260620 06:02:15.387928 7635 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:39867
I20260620 06:02:15.396386 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:02:15.406239 7674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.398364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.3.115.62:34871)
0620 06:02:15.398503 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.398510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.398693 (+ 183us) server_negotiation.cc:407] Connection header received
0620 06:02:15.398855 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.398858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.398911 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.398996 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:15.399711 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.400250 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.401256 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.401486 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.401598 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.403037 (+ 1439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:15.403041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:15.403044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:15.403070 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:15.404503 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:15.404965 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:15.404969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:15.404971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:15.405057 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:15.405342 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:15.405347 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:15.405349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:15.405576 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:15.405721 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.405953 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.406068 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:02:15.409246 7635 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52178:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:15.412375 7635 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:15.426908 8103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.420988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39265 (local address 127.0.0.1:57190)
0620 06:02:15.421816 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.421868 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.422020 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.422815 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.422821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.422853 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:15.423096 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.423106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.424118 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.424124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.425439 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.425449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.426373 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.426376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.426478 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.426513 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.426625 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.426747 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"mutex_wait_us":58,"spinlock_wait_cycles":11648,"thread_start_us":82,"threads_started":1}
I20260620 06:02:15.426910 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.421097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.3.115.2:39265)
0620 06:02:15.421743 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.421755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.422161 (+ 406us) server_negotiation.cc:407] Connection header received
0620 06:02:15.422243 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.422248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.422467 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.422627 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:15.423227 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.423979 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.425576 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.426246 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.426373 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.426527 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.426651 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.426746 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":100,"threads_started":1}
I20260620 06:02:15.427353 8106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.421764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35849 (local address 127.0.0.1:51324)
0620 06:02:15.422062 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.422077 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.422167 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.423007 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.423010 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.423023 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:15.423134 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.423142 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.424585 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.424587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.425668 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.425678 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.426959 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.426962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.427045 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.427067 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.427133 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.427198 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"mutex_wait_us":26,"thread_start_us":102,"threads_started":1}
I20260620 06:02:15.427563 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.421010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40367 (local address 127.0.0.1:56726)
0620 06:02:15.421862 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.421874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.422020 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.422815 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.422821 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.422852 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:15.423045 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.423054 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.424549 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.424552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.425723 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.425731 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.426950 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.426954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.427270 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.427290 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.427356 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.427429 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"mutex_wait_us":62,"spinlock_wait_cycles":10368,"thread_start_us":115,"threads_started":1}
I20260620 06:02:15.427820 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.421973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51324 (local address 127.3.115.1:35849)
0620 06:02:15.422516 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.422523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.422550 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:02:15.422623 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.422628 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.422825 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.423028 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:15.423267 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.424400 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.425797 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.426645 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.426752 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.426886 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.427525 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.427642 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":100,"threads_started":1}
I20260620 06:02:15.428028 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.421250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.3.115.3:40367)
0620 06:02:15.421712 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.421724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.422163 (+ 439us) server_negotiation.cc:407] Connection header received
0620 06:02:15.422243 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.422248 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.422459 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.422624 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:15.423182 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.424402 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.425826 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.426645 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.426752 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.426886 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.427773 (+ 887us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.427883 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":144,"threads_started":1}
I20260620 06:02:15.430581 7892 tablet_service.cc:1467] Processing CreateTablet for tablet c38a23e9b2794c9daa12683209697ee4 (DEFAULT_TABLE table=test-table [id=a4c860ad3bee4543b10554ce7dc229ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:15.430583 7759 tablet_service.cc:1467] Processing CreateTablet for tablet c38a23e9b2794c9daa12683209697ee4 (DEFAULT_TABLE table=test-table [id=a4c860ad3bee4543b10554ce7dc229ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:15.430614 8025 tablet_service.cc:1467] Processing CreateTablet for tablet c38a23e9b2794c9daa12683209697ee4 (DEFAULT_TABLE table=test-table [id=a4c860ad3bee4543b10554ce7dc229ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:15.431612 8025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38a23e9b2794c9daa12683209697ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:15.431618 7759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38a23e9b2794c9daa12683209697ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:15.431622 7892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c38a23e9b2794c9daa12683209697ee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:15.437939 8108 tablet_bootstrap.cc:492] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Bootstrap starting.
I20260620 06:02:15.439322 8109 tablet_bootstrap.cc:492] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: Bootstrap starting.
I20260620 06:02:15.440241 8108 tablet_bootstrap.cc:654] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:15.440963 8110 tablet_bootstrap.cc:492] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Bootstrap starting.
I20260620 06:02:15.441349 8109 tablet_bootstrap.cc:654] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:15.441540 8108 log.cc:824] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:15.442166 8109 log.cc:824] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:15.443053 8110 tablet_bootstrap.cc:654] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:15.443562 8108 tablet_bootstrap.cc:492] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: No bootstrap required, opened a new log
I20260620 06:02:15.443766 8108 ts_tablet_manager.cc:1397] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:02:15.444149 8109 tablet_bootstrap.cc:492] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: No bootstrap required, opened a new log
I20260620 06:02:15.444331 8109 ts_tablet_manager.cc:1397] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:02:15.444360 8110 log.cc:824] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:15.446424 8110 tablet_bootstrap.cc:492] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: No bootstrap required, opened a new log
I20260620 06:02:15.446599 8110 ts_tablet_manager.cc:1397] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:02:15.447098 8108 raft_consensus.cc:348] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.447347 8108 raft_consensus.cc:374] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:15.447419 8108 raft_consensus.cc:729] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158c8b25f1ee4421858fcb3dd76f3870, State: Initialized, Role: FOLLOWER
I20260620 06:02:15.447891 8108 consensus_queue.cc:260] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.448518 8091 heartbeater.cc:503] Master 127.3.115.62:34871 was elected leader, sending a full tablet report...
I20260620 06:02:15.448904 8108 ts_tablet_manager.cc:1428] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:02:15.448745 8109 raft_consensus.cc:348] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.448980 8109 raft_consensus.cc:374] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:15.449035 8109 raft_consensus.cc:729] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 817b1feb67bf4d9fb2696ab7fb9004e1, State: Initialized, Role: FOLLOWER
I20260620 06:02:15.449579 8109 consensus_queue.cc:260] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.450333 7825 heartbeater.cc:503] Master 127.3.115.62:34871 was elected leader, sending a full tablet report...
I20260620 06:02:15.450363 8110 raft_consensus.cc:348] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.450601 8110 raft_consensus.cc:374] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:15.450620 8109 ts_tablet_manager.cc:1428] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:02:15.450655 8110 raft_consensus.cc:729] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25009ccbc95847b3a9af7069627e6757, State: Initialized, Role: FOLLOWER
I20260620 06:02:15.451062 8110 consensus_queue.cc:260] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.451740 7958 heartbeater.cc:503] Master 127.3.115.62:34871 was elected leader, sending a full tablet report...
I20260620 06:02:15.451924 8110 ts_tablet_manager.cc:1428] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 06:02:15.619293 8092 tablet.cc:2369] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:02:15.638396 7959 tablet.cc:2369] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:15.639468 8115 raft_consensus.cc:482] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:15.639679 8115 raft_consensus.cc:504] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.640873 8115 leader_election.cc:288] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25009ccbc95847b3a9af7069627e6757 (127.3.115.2:39265), 158c8b25f1ee4421858fcb3dd76f3870 (127.3.115.3:40367)
I20260620 06:02:15.645601 8104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.641241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:58539 (local address 127.3.115.2:39265)
0620 06:02:15.641382 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.641389 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.641756 (+ 367us) server_negotiation.cc:407] Connection header received
0620 06:02:15.641980 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.641984 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.642044 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.642113 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:15.642446 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.643170 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.644545 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.645033 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.645153 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.645217 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.645401 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.645477 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:02:15.645617 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.641252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39265 (local address 127.3.115.1:58539)
0620 06:02:15.641660 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.641675 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.641830 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.642191 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.642194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.642209 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:15.642328 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.642337 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.643278 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.643283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.644426 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.644433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.645160 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.645164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.645264 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.645286 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.645384 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.645460 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":137,"threads_started":1}
I20260620 06:02:15.645802 8105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.641313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:39715 (local address 127.3.115.3:40367)
0620 06:02:15.641443 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.641449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.641724 (+ 275us) server_negotiation.cc:407] Connection header received
0620 06:02:15.641894 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.641897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.641980 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.642058 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:15.642373 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.642899 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.644471 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.645133 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.645232 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.645291 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.645596 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.645682 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:02:15.646164 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.641182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40367 (local address 127.3.115.1:39715)
0620 06:02:15.641606 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.641623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:15.641777 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:15.642090 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:15.642093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:15.642110 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:15.642252 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.642258 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.643017 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.643022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.644331 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:15.644341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.645349 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.645352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:15.645454 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.645476 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:02:15.645940 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 06:02:15.646016 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":1,"thread_start_us":104,"threads_started":1}
I20260620 06:02:15.646329 7912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38a23e9b2794c9daa12683209697ee4" candidate_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25009ccbc95847b3a9af7069627e6757" is_pre_election: true
I20260620 06:02:15.646667 7912 raft_consensus.cc:2454] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817b1feb67bf4d9fb2696ab7fb9004e1 in term 0.
I20260620 06:02:15.646739 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38a23e9b2794c9daa12683209697ee4" candidate_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158c8b25f1ee4421858fcb3dd76f3870" is_pre_election: true
I20260620 06:02:15.647001 8045 raft_consensus.cc:2454] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 817b1feb67bf4d9fb2696ab7fb9004e1 in term 0.
I20260620 06:02:15.647164 7714 leader_election.cc:302] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25009ccbc95847b3a9af7069627e6757, 817b1feb67bf4d9fb2696ab7fb9004e1; no voters:
I20260620 06:02:15.647508 8115 raft_consensus.cc:2781] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:15.647606 8115 raft_consensus.cc:482] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:15.647650 8115 raft_consensus.cc:3037] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:15.649003 8115 raft_consensus.cc:504] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.649377 8115 leader_election.cc:288] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [CANDIDATE]: Term 1 election: Requested vote from peers 25009ccbc95847b3a9af7069627e6757 (127.3.115.2:39265), 158c8b25f1ee4421858fcb3dd76f3870 (127.3.115.3:40367)
I20260620 06:02:15.649744 8045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38a23e9b2794c9daa12683209697ee4" candidate_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "158c8b25f1ee4421858fcb3dd76f3870"
I20260620 06:02:15.649765 7912 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c38a23e9b2794c9daa12683209697ee4" candidate_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25009ccbc95847b3a9af7069627e6757"
I20260620 06:02:15.649861 8045 raft_consensus.cc:3037] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:15.649911 7912 raft_consensus.cc:3037] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:15.650987 7912 raft_consensus.cc:2454] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817b1feb67bf4d9fb2696ab7fb9004e1 in term 1.
I20260620 06:02:15.651080 8045 raft_consensus.cc:2454] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 817b1feb67bf4d9fb2696ab7fb9004e1 in term 1.
I20260620 06:02:15.651296 7714 leader_election.cc:302] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25009ccbc95847b3a9af7069627e6757, 817b1feb67bf4d9fb2696ab7fb9004e1; no voters:
I20260620 06:02:15.651499 8115 raft_consensus.cc:2781] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:15.651847 8115 raft_consensus.cc:686] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 1 LEADER]: Becoming Leader. State: Replica: 817b1feb67bf4d9fb2696ab7fb9004e1, State: Running, Role: LEADER
I20260620 06:02:15.652457 8115 consensus_queue.cc:237] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } }
I20260620 06:02:15.656018 7634 catalog_manager.cc:5495] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 817b1feb67bf4d9fb2696ab7fb9004e1 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "817b1feb67bf4d9fb2696ab7fb9004e1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35849 } health_report { overall_health: HEALTHY } } }
W20260620 06:02:15.668196 7826 tablet.cc:2369] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:15.719767 8107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.716144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51328 (local address 127.3.115.1:35849)
0620 06:02:15.716282 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.716310 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.716325 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:15.716409 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.716411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.716465 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.716613 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:15.716878 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.717399 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.718267 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:15.718490 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:15.718618 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:15.718936 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:15.719364 (+ 428us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:15.719515 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.719549 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.719617 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:02:15.725889 7912 raft_consensus.cc:1261] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 1 FOLLOWER]: Refusing update from remote peer 817b1feb67bf4d9fb2696ab7fb9004e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:15.726009 8045 raft_consensus.cc:1261] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 1 FOLLOWER]: Refusing update from remote peer 817b1feb67bf4d9fb2696ab7fb9004e1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:15.726634 8115 consensus_queue.cc:1035] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25009ccbc95847b3a9af7069627e6757" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:15.726857 8119 consensus_queue.cc:1035] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "158c8b25f1ee4421858fcb3dd76f3870" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:15.733788 8122 mvcc.cc:205] Tried to move back new op lower bound from 7298807135125323776 to 7298807134836178944. Current Snapshot: MvccSnapshot[applied={T|T < 7298807135125323776}]
I20260620 06:02:15.734007 8124 mvcc.cc:205] Tried to move back new op lower bound from 7298807135125323776 to 7298807134836178944. Current Snapshot: MvccSnapshot[applied={T|T < 7298807135125323776}]
I20260620 06:02:15.735388 8123 mvcc.cc:205] Tried to move back new op lower bound from 7298807135125323776 to 7298807134836178944. Current Snapshot: MvccSnapshot[applied={T|T < 7298807135125323776}]
I20260620 06:02:15.743255 7633 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:15.743448 7633 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52178:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:15.745957 7633 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00f16ef762944128b8fdf852735f16ba: Sending DeleteTablet for 3 replicas of tablet c38a23e9b2794c9daa12683209697ee4
I20260620 06:02:15.746765 7892 tablet_service.cc:1514] Processing DeleteTablet for tablet c38a23e9b2794c9daa12683209697ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:15 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:57190
I20260620 06:02:15.746866 7759 tablet_service.cc:1514] Processing DeleteTablet for tablet c38a23e9b2794c9daa12683209697ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:15 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:51324
I20260620 06:02:15.746838 8025 tablet_service.cc:1514] Processing DeleteTablet for tablet c38a23e9b2794c9daa12683209697ee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:15 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:56726
I20260620 06:02:15.747334 8132 tablet_replica.cc:330] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1: stopping tablet replica
I20260620 06:02:15.747604 8131 tablet_replica.cc:330] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: stopping tablet replica
I20260620 06:02:15.747653 8132 raft_consensus.cc:2229] T c38a23e9b2794c9daa12683209697ee4 P 817b1feb67bf4d9fb2696ab7fb9004e1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:15.747743 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7697
I20260620 06:02:15.748719 8131 raft_consensus.cc:2229] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:15.749059 8131 raft_consensus.cc:2258] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:15.749830 8133 tablet_replica.cc:330] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: stopping tablet replica
I20260620 06:02:15.750110 8133 raft_consensus.cc:2229] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:15.750384 8133 raft_consensus.cc:2258] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:15.750525 8131 ts_tablet_manager.cc:1905] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:15.751765 8133 ts_tablet_manager.cc:1905] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:15.753147 8131 ts_tablet_manager.cc:1918] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:15.753259 8131 log.cc:1196] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/c38a23e9b2794c9daa12683209697ee4
I20260620 06:02:15.753582 8131 ts_tablet_manager.cc:1939] T c38a23e9b2794c9daa12683209697ee4 P 25009ccbc95847b3a9af7069627e6757: Deleting consensus metadata
I20260620 06:02:15.754746 7621 catalog_manager.cc:4841] TS 25009ccbc95847b3a9af7069627e6757 (127.3.115.2:39265): tablet c38a23e9b2794c9daa12683209697ee4 (table test-table [id=a4c860ad3bee4543b10554ce7dc229ab]) successfully deleted
I20260620 06:02:15.755278 8133 ts_tablet_manager.cc:1918] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:15.755385 8133 log.cc:1196] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/c38a23e9b2794c9daa12683209697ee4
I20260620 06:02:15.755723 8133 ts_tablet_manager.cc:1939] T c38a23e9b2794c9daa12683209697ee4 P 158c8b25f1ee4421858fcb3dd76f3870: Deleting consensus metadata
W20260620 06:02:15.756398 7620 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:35849 (error 108)
I20260620 06:02:15.756580 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7830
I20260620 06:02:15.756964 7622 catalog_manager.cc:4841] TS 158c8b25f1ee4421858fcb3dd76f3870 (127.3.115.3:40367): tablet c38a23e9b2794c9daa12683209697ee4 (table test-table [id=a4c860ad3bee4543b10554ce7dc229ab]) successfully deleted
I20260620 06:02:15.757962 8102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.757626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35849 (local address 127.0.0.1:51342)
0620 06:02:15.757779 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:15.757860 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 06:02:15.758152 7620 catalog_manager.cc:4584] TS 817b1feb67bf4d9fb2696ab7fb9004e1 (127.3.115.1:35849): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c38a23e9b2794c9daa12683209697ee4: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35849: connect: Connection refused (error 111)
I20260620 06:02:15.762881 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7963
I20260620 06:02:15.769084 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 7604
2026-06-20T06:02:15Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2481 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T06:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:15Z Disabled control of system clock
I20260620 06:02:15.799858 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:36853
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:36853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:02:15.913223 8138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:15.913492 8138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:15.913532 8138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:15.917565 8138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:15.917621 8138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:15.917636 8138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:15.917652 8138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:15.917668 8138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:15.922312 8138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:36853
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:36853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:15.923336 8138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:15.924790 8138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:15.933182 8146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:15.933192 8144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:15.933199 8143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:15.933544 8138 server_base.cc:1034] running on GCE node
I20260620 06:02:15.934104 8138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:15.935186 8138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:15.936379 8138 hybrid_clock.cc:648] HybridClock initialized: now 1781935335936372 us; error 31 us; skew 500 ppm
I20260620 06:02:15.938232 8138 webserver.cc:460] Webserver started at http://127.3.115.62:37787/ using document root <none> and password file <none>
I20260620 06:02:15.938833 8138 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:15.938925 8138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:15.939167 8138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:15.941254 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "92d7a9b9636e47749d4699573260587a"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "34ed95b5cbb9c138a03a79c383a5bfd9"
server_key_iv: "b78c8900dde94ae99ed102f279e618e3"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.941921 8138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "92d7a9b9636e47749d4699573260587a"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "34ed95b5cbb9c138a03a79c383a5bfd9"
server_key_iv: "b78c8900dde94ae99ed102f279e618e3"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.945644 8138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:02:15.947960 8152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:15.949246 8138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:15.949379 8138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "92d7a9b9636e47749d4699573260587a"
format_stamp: "Formatted at 2026-06-20 06:02:15 on dist-test-slave-4h14"
server_key: "34ed95b5cbb9c138a03a79c383a5bfd9"
server_key_iv: "b78c8900dde94ae99ed102f279e618e3"
server_key_version: "encryptionkey@0"
I20260620 06:02:15.949488 8138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:15.953918 8138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:15.954499 8138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:15.954690 8138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:15.961467 8138 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:36853
I20260620 06:02:15.961488 8204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:36853 every 8 connection(s)
I20260620 06:02:15.962378 8138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:15.965294 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8138
I20260620 06:02:15.965418 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:15.965760 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:15.965785 3532 external_mini_cluster.cc:1428] Setting key 1ec7bf9fe193eb128a1053e9a98f95f3
I20260620 06:02:15.971101 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:15.967035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.3.115.62:36853)
0620 06:02:15.967622 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:15.967636 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:15.967671 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:02:15.968314 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:15.968331 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:15.968740 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:15.969084 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:15.969856 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:15.969860 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:15.969871 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:15.969881 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:15.969934 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:15.969950 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:15.970136 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:15.970210 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 06:02:15.970437 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
I20260620 06:02:15.972739 8205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a: Bootstrap starting.
I20260620 06:02:15.975119 8205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:15.975824 8205 log.cc:824] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:15.977738 8205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a: No bootstrap required, opened a new log
I20260620 06:02:15.981451 8205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7a9b9636e47749d4699573260587a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36853 } }
I20260620 06:02:15.981678 8205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:15.981721 8205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d7a9b9636e47749d4699573260587a, State: Initialized, Role: FOLLOWER
I20260620 06:02:15.982089 8205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7a9b9636e47749d4699573260587a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36853 } }
I20260620 06:02:15.982239 8205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:15.982300 8205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:15.982374 8205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:15.983223 8205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7a9b9636e47749d4699573260587a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36853 } }
I20260620 06:02:15.983606 8205 leader_election.cc:302] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92d7a9b9636e47749d4699573260587a; no voters:
I20260620 06:02:15.983985 8205 leader_election.cc:288] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:15.984071 8210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:15.984313 8210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [term 1 LEADER]: Becoming Leader. State: Replica: 92d7a9b9636e47749d4699573260587a, State: Running, Role: LEADER
I20260620 06:02:15.984627 8210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7a9b9636e47749d4699573260587a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36853 } }
I20260620 06:02:15.985374 8205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:15.986326 8211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92d7a9b9636e47749d4699573260587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7a9b9636e47749d4699573260587a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36853 } } }
I20260620 06:02:15.986363 8212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92d7a9b9636e47749d4699573260587a. Latest consensus state: current_term: 1 leader_uuid: "92d7a9b9636e47749d4699573260587a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92d7a9b9636e47749d4699573260587a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36853 } } }
I20260620 06:02:15.986485 8212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:15.986483 8211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:15.986944 8219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:15.990648 8219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:15.996585 8219 catalog_manager.cc:1347] Generated new cluster ID: 2c1fede6649e42abbc874fe5484072b6
I20260620 06:02:15.996791 8219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:16.008131 8219 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:16.008462 8219 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:16.019526 8219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a: Generated new TSK 0
I20260620 06:02:16.020215 8219 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:16.028941 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36853
--builtin_ntp_servers=127.3.115.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:02:16.147861 8229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:16.148146 8229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:16.148190 8229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:16.152441 8229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:16.152513 8229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:16.152572 8229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:16.157891 8229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:16.158980 8229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:16.160358 8229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:16.170398 8237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:16.170405 8234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.170671 8229 server_base.cc:1034] running on GCE node
W20260620 06:02:16.170409 8235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.171197 8229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:16.171789 8229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:16.172940 8229 hybrid_clock.cc:648] HybridClock initialized: now 1781935336172891 us; error 63 us; skew 500 ppm
I20260620 06:02:16.174853 8229 webserver.cc:460] Webserver started at http://127.3.115.1:38333/ using document root <none> and password file <none>
I20260620 06:02:16.175469 8229 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:16.175534 8229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:16.175704 8229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:16.177799 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "0a5013e02af94b678929e6750c107aaa"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "16c65cbd1c000ede1c871d77043a99e7"
server_key_iv: "99c8495b8f56f8122a0109925cb6b322"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.178324 8229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "0a5013e02af94b678929e6750c107aaa"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "16c65cbd1c000ede1c871d77043a99e7"
server_key_iv: "99c8495b8f56f8122a0109925cb6b322"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.181906 8229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:02:16.184396 8243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.185863 8229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:16.186017 8229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "0a5013e02af94b678929e6750c107aaa"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "16c65cbd1c000ede1c871d77043a99e7"
server_key_iv: "99c8495b8f56f8122a0109925cb6b322"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.186151 8229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:16.190811 8229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:16.191425 8229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:16.191610 8229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:16.192179 8229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:16.193146 8229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:16.193223 8229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.193312 8229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:16.193363 8229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.202437 8229 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:42813
I20260620 06:02:16.202459 8356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:42813 every 8 connection(s)
I20260620 06:02:16.203366 8229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:16.205655 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8229
I20260620 06:02:16.205804 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:16.206125 3532 external_mini_cluster.cc:1428] Setting key 3cec7697362a24f436ad375d2e10b3cd
I20260620 06:02:16.208061 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36853
--builtin_ntp_servers=127.3.115.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260620 06:02:16.209790 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.205223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:60783 (local address 127.3.115.62:36853)
0620 06:02:16.205354 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.205358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.206555 (+ 1197us) server_negotiation.cc:407] Connection header received
0620 06:02:16.207993 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.207997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.208036 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.208102 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.209356 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.209359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.209361 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.209372 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.209392 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.209400 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.209480 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.209653 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.209689 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:02:16.210951 8359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.205643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36853 (local address 127.3.115.1:60783)
0620 06:02:16.206388 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.206440 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.207791 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.208292 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.208301 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.208419 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.209041 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.209043 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.209083 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.209203 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.209491 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.209501 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.209507 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.209685 (+ 178us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.210049 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":165,"threads_started":1}
I20260620 06:02:16.212261 8357 heartbeater.cc:347] Connected to a master server at 127.3.115.62:36853
I20260620 06:02:16.212486 8357 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:16.212941 8357 heartbeater.cc:511] Master 127.3.115.62:36853 requested a full tablet report, sending...
I20260620 06:02:16.214725 8169 ts_manager.cc:194] Registered new tserver with Master: 0a5013e02af94b678929e6750c107aaa (127.3.115.1:42813)
W20260620 06:02:16.321151 8360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:16.321427 8360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:16.321467 8360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:16.325506 8360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:16.325567 8360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:16.325623 8360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:16.330371 8360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:16.331511 8360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:16.332947 8360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:16.342491 8368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:16.342620 8366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.342747 8360 server_base.cc:1034] running on GCE node
W20260620 06:02:16.342499 8365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.343199 8360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:16.343868 8360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:16.345055 8360 hybrid_clock.cc:648] HybridClock initialized: now 1781935336345038 us; error 34 us; skew 500 ppm
I20260620 06:02:16.346983 8360 webserver.cc:460] Webserver started at http://127.3.115.2:42853/ using document root <none> and password file <none>
I20260620 06:02:16.347580 8360 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:16.347645 8360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:16.347857 8360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:16.349968 8360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "8d46da7bf6144d7a8a4903cb6949ecd8"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "429199c44bd2ed0753914f1876afb04b"
server_key_iv: "cf3f2722fd590582fff9fe9477a68685"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.350529 8360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "8d46da7bf6144d7a8a4903cb6949ecd8"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "429199c44bd2ed0753914f1876afb04b"
server_key_iv: "cf3f2722fd590582fff9fe9477a68685"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.354633 8360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:16.357224 8374 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.358486 8360 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:16.358626 8360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "8d46da7bf6144d7a8a4903cb6949ecd8"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "429199c44bd2ed0753914f1876afb04b"
server_key_iv: "cf3f2722fd590582fff9fe9477a68685"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.358748 8360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:16.363431 8360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:16.364043 8360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:16.364243 8360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:16.364794 8360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:16.365633 8360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:16.365679 8360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.365716 8360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:16.365731 8360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.373843 8360 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:42347
I20260620 06:02:16.373896 8487 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:42347 every 8 connection(s)
I20260620 06:02:16.374758 8360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:16.380649 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.376624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:54913 (local address 127.3.115.62:36853)
0620 06:02:16.376796 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.376800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.377848 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 06:02:16.379113 (+ 1265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.379115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.379152 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.379230 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.380255 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.380258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.380260 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.380271 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.380291 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.380299 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.380376 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.380485 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.380521 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 06:02:16.381526 8490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.377053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36853 (local address 127.3.115.2:54913)
0620 06:02:16.377702 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.377742 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.378920 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.379389 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.379396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.379483 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.380017 (+ 534us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.380018 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.380057 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.380121 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.380372 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.380378 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.380382 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.380505 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.380770 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":100,"threads_started":1}
I20260620 06:02:16.382665 8488 heartbeater.cc:347] Connected to a master server at 127.3.115.62:36853
I20260620 06:02:16.382894 8488 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:16.383294 8488 heartbeater.cc:511] Master 127.3.115.62:36853 requested a full tablet report, sending...
I20260620 06:02:16.384356 8169 ts_manager.cc:194] Registered new tserver with Master: 8d46da7bf6144d7a8a4903cb6949ecd8 (127.3.115.2:42347)
I20260620 06:02:16.384616 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8360
I20260620 06:02:16.384789 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:16.385102 3532 external_mini_cluster.cc:1428] Setting key 68bbb3ee61f8c72d79bb65325c859a61
I20260620 06:02:16.386976 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36853
--builtin_ntp_servers=127.3.115.20:37259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 06:02:16.501381 8491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:16.501674 8491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:16.501756 8491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:16.505892 8491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:16.505970 8491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:16.506062 8491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:16.510783 8491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:36853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:16.511960 8491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:16.513506 8491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:16.522943 8497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:16.522933 8499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:16.522950 8496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.523339 8491 server_base.cc:1034] running on GCE node
I20260620 06:02:16.523811 8491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:16.524516 8491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:16.525723 8491 hybrid_clock.cc:648] HybridClock initialized: now 1781935336525701 us; error 37 us; skew 500 ppm
I20260620 06:02:16.527722 8491 webserver.cc:460] Webserver started at http://127.3.115.3:39803/ using document root <none> and password file <none>
I20260620 06:02:16.528321 8491 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:16.528415 8491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:16.528698 8491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:16.530905 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "ed1fbbcc085440b1953769a8c5e13878"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "0a46e429c7c296fbfaf20c834ac67fdd"
server_key_iv: "f39145e55f9da6db8cb418d738447abb"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.531584 8491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "ed1fbbcc085440b1953769a8c5e13878"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "0a46e429c7c296fbfaf20c834ac67fdd"
server_key_iv: "f39145e55f9da6db8cb418d738447abb"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.535429 8491 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:16.537928 8505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.539295 8491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:16.539424 8491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "ed1fbbcc085440b1953769a8c5e13878"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "0a46e429c7c296fbfaf20c834ac67fdd"
server_key_iv: "f39145e55f9da6db8cb418d738447abb"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.539537 8491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:16.544313 8491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:16.544962 8491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:16.545141 8491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:16.545677 8491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:16.546590 8491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:16.546661 8491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.546730 8491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:16.546779 8491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.555896 8491 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40577
I20260620 06:02:16.555915 8618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40577 every 8 connection(s)
I20260620 06:02:16.556898 8491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:16.562994 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.558759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:49131 (local address 127.3.115.62:36853)
0620 06:02:16.558899 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.558903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.560024 (+ 1121us) server_negotiation.cc:407] Connection header received
0620 06:02:16.561368 (+ 1344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.561371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.561409 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.561485 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.562530 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.562533 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.562535 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.562546 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.562624 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.562639 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.562737 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.562850 (+ 113us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.562887 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:02:16.563519 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8491
I20260620 06:02:16.563665 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:16.563920 3532 external_mini_cluster.cc:1428] Setting key 206cce03ede8bcd1d0d826a960ec55f7
I20260620 06:02:16.563982 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.559216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36853 (local address 127.3.115.3:49131)
0620 06:02:16.559874 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.559914 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.561173 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.561661 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.561668 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.561760 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.562288 (+ 528us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.562290 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.562330 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.562398 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.562726 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.562732 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.562737 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.562871 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.563065 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":105,"threads_started":1}
I20260620 06:02:16.565376 8619 heartbeater.cc:347] Connected to a master server at 127.3.115.62:36853
I20260620 06:02:16.565623 8619 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:16.566071 8619 heartbeater.cc:511] Master 127.3.115.62:36853 requested a full tablet report, sending...
I20260620 06:02:16.567011 8168 ts_manager.cc:194] Registered new tserver with Master: ed1fbbcc085440b1953769a8c5e13878 (127.3.115.3:40577)
I20260620 06:02:16.578264 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:02:16.581894 8208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.580296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.3.115.62:36853)
0620 06:02:16.580431 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.580435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.580625 (+ 190us) server_negotiation.cc:407] Connection header received
0620 06:02:16.580783 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.580786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.580824 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.580899 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.581583 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.581586 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.581588 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.581599 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.581618 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.581626 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.581699 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.581760 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.581792 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:02:16.584791 8168 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:16.588030 8168 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:16.600417 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.596824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40577 (local address 127.0.0.1:41054)
0620 06:02:16.597820 (+ 996us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.597870 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.597992 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.598911 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.598914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.598930 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.599522 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.599524 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.599544 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.599611 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.600152 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.600155 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.600158 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.600228 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.600262 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"thread_start_us":72,"threads_started":1}
I20260620 06:02:16.600883 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.596960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41054 (local address 127.3.115.3:40577)
0620 06:02:16.598573 (+ 1613us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.598581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.598598 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:02:16.598648 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.598652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.598785 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.598924 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.599803 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.599811 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.599843 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.599868 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.599960 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.600000 (+ 40us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.600113 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.600641 (+ 528us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.600762 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1532,"thread_start_us":97,"threads_started":1}
I20260620 06:02:16.601068 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.597801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47648 (local address 127.3.115.1:42813)
0620 06:02:16.598142 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.598152 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.598394 (+ 242us) server_negotiation.cc:407] Connection header received
0620 06:02:16.598446 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.598450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.598627 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.598778 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.599860 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.599866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.599877 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.599889 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.599935 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.599967 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.600768 (+ 801us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.600815 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.600886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":120,"threads_started":1}
I20260620 06:02:16.600420 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.596978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:42813 (local address 127.0.0.1:47648)
0620 06:02:16.598291 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.598304 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.598387 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.598777 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.598783 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.598862 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.599676 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.599678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.599693 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.599719 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.600076 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.600087 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.600095 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.600186 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.600252 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1247,"thread_start_us":125,"threads_started":1}
I20260620 06:02:16.601917 8630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.597408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42347 (local address 127.0.0.1:47046)
0620 06:02:16.599596 (+ 2188us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.599619 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.599697 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.600787 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.600790 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.600804 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.601327 (+ 523us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.601329 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.601340 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.601367 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.601687 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.601691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.601694 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.601781 (+ 87us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.601814 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"thread_start_us":100,"threads_started":1}
I20260620 06:02:16.601984 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.597949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47046 (local address 127.3.115.2:42347)
0620 06:02:16.598254 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.598270 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.599702 (+ 1432us) server_negotiation.cc:407] Connection header received
0620 06:02:16.599830 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.599837 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.600018 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.600157 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.601489 (+ 1332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.601495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.601506 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.601519 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.601574 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.601598 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.601682 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.601784 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.601835 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":102,"threads_started":1}
I20260620 06:02:16.604434 8553 tablet_service.cc:1467] Processing CreateTablet for tablet af04e911663f45d48c8bb08079b75cb4 (DEFAULT_TABLE table=test-table [id=771ca3f6532b4b6c9b5039cd61eed00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:16.604619 8422 tablet_service.cc:1467] Processing CreateTablet for tablet af04e911663f45d48c8bb08079b75cb4 (DEFAULT_TABLE table=test-table [id=771ca3f6532b4b6c9b5039cd61eed00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:16.605558 8422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af04e911663f45d48c8bb08079b75cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:16.605574 8553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af04e911663f45d48c8bb08079b75cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:16.605191 8291 tablet_service.cc:1467] Processing CreateTablet for tablet af04e911663f45d48c8bb08079b75cb4 (DEFAULT_TABLE table=test-table [id=771ca3f6532b4b6c9b5039cd61eed00c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:16.606104 8291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af04e911663f45d48c8bb08079b75cb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:16.612531 8636 tablet_bootstrap.cc:492] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Bootstrap starting.
I20260620 06:02:16.614149 8638 tablet_bootstrap.cc:492] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Bootstrap starting.
I20260620 06:02:16.614646 8636 tablet_bootstrap.cc:654] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:16.615371 8637 tablet_bootstrap.cc:492] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: Bootstrap starting.
I20260620 06:02:16.615717 8636 log.cc:824] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:16.616608 8638 tablet_bootstrap.cc:654] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:16.617463 8638 log.cc:824] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:16.617729 8637 tablet_bootstrap.cc:654] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:16.618572 8636 tablet_bootstrap.cc:492] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: No bootstrap required, opened a new log
I20260620 06:02:16.618610 8637 log.cc:824] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:16.618786 8636 ts_tablet_manager.cc:1397] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:02:16.620697 8637 tablet_bootstrap.cc:492] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: No bootstrap required, opened a new log
I20260620 06:02:16.620896 8637 ts_tablet_manager.cc:1397] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:02:16.622731 8636 raft_consensus.cc:348] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.622970 8636 raft_consensus.cc:374] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:16.623023 8636 raft_consensus.cc:729] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d46da7bf6144d7a8a4903cb6949ecd8, State: Initialized, Role: FOLLOWER
I20260620 06:02:16.623467 8636 consensus_queue.cc:260] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.624990 8636 ts_tablet_manager.cc:1428] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:02:16.625360 8488 heartbeater.cc:503] Master 127.3.115.62:36853 was elected leader, sending a full tablet report...
W20260620 06:02:16.625697 8489 tablet.cc:2369] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:16.626793 8638 tablet_bootstrap.cc:492] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: No bootstrap required, opened a new log
I20260620 06:02:16.626988 8638 ts_tablet_manager.cc:1397] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20260620 06:02:16.627147 8637 raft_consensus.cc:348] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.627394 8637 raft_consensus.cc:374] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:16.627454 8637 raft_consensus.cc:729] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a5013e02af94b678929e6750c107aaa, State: Initialized, Role: FOLLOWER
I20260620 06:02:16.627889 8637 consensus_queue.cc:260] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.628504 8357 heartbeater.cc:503] Master 127.3.115.62:36853 was elected leader, sending a full tablet report...
I20260620 06:02:16.629024 8637 ts_tablet_manager.cc:1428] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260620 06:02:16.631235 8638 raft_consensus.cc:348] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.631492 8638 raft_consensus.cc:374] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:16.631554 8638 raft_consensus.cc:729] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1fbbcc085440b1953769a8c5e13878, State: Initialized, Role: FOLLOWER
I20260620 06:02:16.631987 8638 consensus_queue.cc:260] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.632587 8619 heartbeater.cc:503] Master 127.3.115.62:36853 was elected leader, sending a full tablet report...
I20260620 06:02:16.632946 8638 ts_tablet_manager.cc:1428] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Time spent starting tablet: real 0.006s user 0.004s sys 0.003s
I20260620 06:02:16.700073 8644 raft_consensus.cc:482] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:16.700310 8644 raft_consensus.cc:504] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.701499 8644 leader_election.cc:288] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a5013e02af94b678929e6750c107aaa (127.3.115.1:42813), 8d46da7bf6144d7a8a4903cb6949ecd8 (127.3.115.2:42347)
I20260620 06:02:16.703442 8621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.701754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42347 (local address 127.3.115.3:46471)
0620 06:02:16.701866 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.701883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.701973 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.702299 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.702303 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.702319 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.702867 (+ 548us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.702871 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.702884 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.702917 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.703192 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.703195 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.703198 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.703275 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.703312 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 06:02:16.703442 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.701866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:46471 (local address 127.3.115.2:42347)
0620 06:02:16.702006 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.702011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.702049 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:02:16.702141 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.702143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.702181 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.702249 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.703058 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.703061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.703063 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.703075 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.703095 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.703103 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.703209 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.703273 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.703312 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:02:16.703874 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.701761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:42813 (local address 127.3.115.3:38349)
0620 06:02:16.702290 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.702304 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:16.702392 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:16.702679 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:16.702681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:16.702693 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:16.703207 (+ 514us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:16.703209 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:16.703217 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:16.703242 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:16.703639 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:16.703642 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:16.703645 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 06:02:16.703726 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:02:16.703759 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":51,"thread_start_us":134,"threads_started":1}
I20260620 06:02:16.703924 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.701995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38349 (local address 127.3.115.1:42813)
0620 06:02:16.702152 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.702156 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.702396 (+ 240us) server_negotiation.cc:407] Connection header received
0620 06:02:16.702475 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.702477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.702541 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.702655 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.703367 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.703373 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.703380 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.703391 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.703410 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.703418 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.703511 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.703761 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.703809 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:02:16.704274 8442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af04e911663f45d48c8bb08079b75cb4" candidate_uuid: "ed1fbbcc085440b1953769a8c5e13878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" is_pre_election: true
W20260620 06:02:16.704485 8358 tablet.cc:2369] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:16.704528 8311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af04e911663f45d48c8bb08079b75cb4" candidate_uuid: "ed1fbbcc085440b1953769a8c5e13878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a5013e02af94b678929e6750c107aaa" is_pre_election: true
I20260620 06:02:16.704625 8442 raft_consensus.cc:2454] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1fbbcc085440b1953769a8c5e13878 in term 0.
I20260620 06:02:16.704831 8311 raft_consensus.cc:2454] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed1fbbcc085440b1953769a8c5e13878 in term 0.
I20260620 06:02:16.705184 8508 leader_election.cc:302] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d46da7bf6144d7a8a4903cb6949ecd8, ed1fbbcc085440b1953769a8c5e13878; no voters:
I20260620 06:02:16.705457 8644 raft_consensus.cc:2781] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:16.705559 8644 raft_consensus.cc:482] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:16.705616 8644 raft_consensus.cc:3037] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:16.706704 8644 raft_consensus.cc:504] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.707085 8644 leader_election.cc:288] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [CANDIDATE]: Term 1 election: Requested vote from peers 0a5013e02af94b678929e6750c107aaa (127.3.115.1:42813), 8d46da7bf6144d7a8a4903cb6949ecd8 (127.3.115.2:42347)
I20260620 06:02:16.707418 8311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af04e911663f45d48c8bb08079b75cb4" candidate_uuid: "ed1fbbcc085440b1953769a8c5e13878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a5013e02af94b678929e6750c107aaa"
I20260620 06:02:16.707507 8442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af04e911663f45d48c8bb08079b75cb4" candidate_uuid: "ed1fbbcc085440b1953769a8c5e13878" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8"
I20260620 06:02:16.707557 8311 raft_consensus.cc:3037] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:16.707690 8442 raft_consensus.cc:3037] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:16.708632 8311 raft_consensus.cc:2454] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1fbbcc085440b1953769a8c5e13878 in term 1.
I20260620 06:02:16.708804 8442 raft_consensus.cc:2454] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed1fbbcc085440b1953769a8c5e13878 in term 1.
I20260620 06:02:16.708992 8506 leader_election.cc:302] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a5013e02af94b678929e6750c107aaa, ed1fbbcc085440b1953769a8c5e13878; no voters:
I20260620 06:02:16.709174 8644 raft_consensus.cc:2781] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:16.709410 8644 raft_consensus.cc:686] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 1 LEADER]: Becoming Leader. State: Replica: ed1fbbcc085440b1953769a8c5e13878, State: Running, Role: LEADER
I20260620 06:02:16.709756 8644 consensus_queue.cc:237] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } }
I20260620 06:02:16.712894 8169 catalog_manager.cc:5495] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed1fbbcc085440b1953769a8c5e13878 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "ed1fbbcc085440b1953769a8c5e13878" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1fbbcc085440b1953769a8c5e13878" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40577 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 } health_report { overall_health: UNKNOWN } } }
I20260620 06:02:16.721621 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.719989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41064 (local address 127.3.115.3:40577)
0620 06:02:16.720116 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.720120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.720138 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:02:16.720282 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.720284 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.720338 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.720406 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.721038 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.721041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.721043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.721053 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.721070 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.721079 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.721361 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.721479 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.721514 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:02:16.727489 8442 raft_consensus.cc:1261] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 1 FOLLOWER]: Refusing update from remote peer ed1fbbcc085440b1953769a8c5e13878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:16.727489 8311 raft_consensus.cc:1261] T af04e911663f45d48c8bb08079b75cb4 P 0a5013e02af94b678929e6750c107aaa [term 1 FOLLOWER]: Refusing update from remote peer ed1fbbcc085440b1953769a8c5e13878: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:16.728250 8644 consensus_queue.cc:1035] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d46da7bf6144d7a8a4903cb6949ecd8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:16.728497 8646 consensus_queue.cc:1035] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a5013e02af94b678929e6750c107aaa" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:16.736294 8649 mvcc.cc:205] Tried to move back new op lower bound from 7298807139228340224 to 7298807139165761536. Current Snapshot: MvccSnapshot[applied={T|T < 7298807139228340224}]
I20260620 06:02:16.736707 8651 mvcc.cc:205] Tried to move back new op lower bound from 7298807139228340224 to 7298807139165761536. Current Snapshot: MvccSnapshot[applied={T|T < 7298807139228340224}]
I20260620 06:02:16.737365 8650 mvcc.cc:205] Tried to move back new op lower bound from 7298807139228340224 to 7298807139165761536. Current Snapshot: MvccSnapshot[applied={T|T < 7298807139228340224}]
I20260620 06:02:16.745827 8168 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:16.746021 8168 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:16.748538 8168 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92d7a9b9636e47749d4699573260587a: Sending DeleteTablet for 3 replicas of tablet af04e911663f45d48c8bb08079b75cb4
I20260620 06:02:16.749369 8422 tablet_service.cc:1514] Processing DeleteTablet for tablet af04e911663f45d48c8bb08079b75cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:16 UTC) from {username='slave'} at 127.0.0.1:47046
I20260620 06:02:16.749361 8553 tablet_service.cc:1514] Processing DeleteTablet for tablet af04e911663f45d48c8bb08079b75cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:16 UTC) from {username='slave'} at 127.0.0.1:41054
I20260620 06:02:16.749394 8291 tablet_service.cc:1514] Processing DeleteTablet for tablet af04e911663f45d48c8bb08079b75cb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:16 UTC) from {username='slave'} at 127.0.0.1:47648
I20260620 06:02:16.750016 8660 tablet_replica.cc:330] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: stopping tablet replica
I20260620 06:02:16.750332 8660 raft_consensus.cc:2229] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:16.750627 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8229
I20260620 06:02:16.750758 8660 raft_consensus.cc:2258] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:16.751119 8659 tablet_replica.cc:330] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: stopping tablet replica
I20260620 06:02:16.751477 8659 raft_consensus.cc:2229] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:16.751785 8659 raft_consensus.cc:2258] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:16.753965 8660 ts_tablet_manager.cc:1905] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:16.754272 8659 ts_tablet_manager.cc:1905] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:16.756340 8659 ts_tablet_manager.cc:1918] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:16.756459 8659 log.cc:1196] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/af04e911663f45d48c8bb08079b75cb4
I20260620 06:02:16.756804 8659 ts_tablet_manager.cc:1939] T af04e911663f45d48c8bb08079b75cb4 P 8d46da7bf6144d7a8a4903cb6949ecd8: Deleting consensus metadata
I20260620 06:02:16.756790 8660 ts_tablet_manager.cc:1918] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:16.756886 8660 log.cc:1196] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/af04e911663f45d48c8bb08079b75cb4
I20260620 06:02:16.757210 8660 ts_tablet_manager.cc:1939] T af04e911663f45d48c8bb08079b75cb4 P ed1fbbcc085440b1953769a8c5e13878: Deleting consensus metadata
I20260620 06:02:16.757973 8155 catalog_manager.cc:4841] TS 8d46da7bf6144d7a8a4903cb6949ecd8 (127.3.115.2:42347): tablet af04e911663f45d48c8bb08079b75cb4 (table test-table [id=771ca3f6532b4b6c9b5039cd61eed00c]) successfully deleted
I20260620 06:02:16.758240 8154 catalog_manager.cc:4841] TS ed1fbbcc085440b1953769a8c5e13878 (127.3.115.3:40577): tablet af04e911663f45d48c8bb08079b75cb4 (table test-table [id=771ca3f6532b4b6c9b5039cd61eed00c]) successfully deleted
W20260620 06:02:16.765206 8153 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:42813 (error 108)
I20260620 06:02:16.765264 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8360
I20260620 06:02:16.767666 8632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.767365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:42813 (local address 127.0.0.1:47652)
0620 06:02:16.767497 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:16.767576 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:42813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 06:02:16.767854 8153 catalog_manager.cc:4584] TS 0a5013e02af94b678929e6750c107aaa (127.3.115.1:42813): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af04e911663f45d48c8bb08079b75cb4: Network error: Client connection negotiation failed: client connection to 127.3.115.1:42813: connect: Connection refused (error 111)
I20260620 06:02:16.772192 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8491
I20260620 06:02:16.778110 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8138
2026-06-20T06:02:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1007 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 06:02:16.799515 3532 mini_oidc.cc:152] Starting JWKS server
I20260620 06:02:16.801350 3532 webserver.cc:460] Webserver started at https://127.0.0.1:37401/ using document root <none> and password file <none>
I20260620 06:02:16.801549 3532 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:02:16.801827 3532 webserver.cc:460] Webserver started at http://127.0.0.1:43421/ using document root <none> and password file <none>
2026-06-20T06:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:16Z Disabled control of system clock
I20260620 06:02:16.810863 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:46103
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:39785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:46103
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37401/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/testchainca.cert with env {}
W20260620 06:02:16.921401 8668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:16.921674 8668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:16.921715 8668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:16.925760 8668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:16.925815 8668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:16.925830 8668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:16.925846 8668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:16.925861 8668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:02:16.925877 8668 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:02:16.925890 8668 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37401/jwks.json
I20260620 06:02:16.931142 8668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:46103
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:46103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37401/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:16.932197 8668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:16.933255 8668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:16.941944 8674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:16.941937 8676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.942303 8668 server_base.cc:1034] running on GCE node
W20260620 06:02:16.941937 8673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:16.943087 8668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:16.944435 8668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:16.945636 8668 hybrid_clock.cc:648] HybridClock initialized: now 1781935336945631 us; error 57 us; skew 500 ppm
I20260620 06:02:16.947679 8668 webserver.cc:460] Webserver started at http://127.3.115.62:37243/ using document root <none> and password file <none>
I20260620 06:02:16.948309 8668 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:16.948411 8668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:16.948633 8668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:16.950798 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "372d40ef17e94e2fb4a15ad01c256a6c"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "59e88b70f725436b41b60c545a79924b"
server_key_iv: "befe46f1ea11204a2f96866fe375a6e2"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.951390 8668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "372d40ef17e94e2fb4a15ad01c256a6c"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "59e88b70f725436b41b60c545a79924b"
server_key_iv: "befe46f1ea11204a2f96866fe375a6e2"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.955279 8668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:16.957816 8682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:16.959019 8668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:16.959165 8668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "372d40ef17e94e2fb4a15ad01c256a6c"
format_stamp: "Formatted at 2026-06-20 06:02:16 on dist-test-slave-4h14"
server_key: "59e88b70f725436b41b60c545a79924b"
server_key_iv: "befe46f1ea11204a2f96866fe375a6e2"
server_key_version: "encryptionkey@0"
I20260620 06:02:16.959280 8668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:16.976974 8668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:16.977682 8668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:16.977882 8668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:16.984956 8668 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:46103
I20260620 06:02:16.984961 8734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:46103 every 8 connection(s)
I20260620 06:02:16.985937 8668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:16.986714 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8668
I20260620 06:02:16.986871 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:16.987200 3532 external_mini_cluster.cc:1428] Setting key 73c2a15add0f69416b9c267e7053b861
I20260620 06:02:16.989782 8735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:16.995514 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c: Bootstrap starting.
I20260620 06:02:16.996551 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:16.988532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36458 (local address 127.3.115.62:46103)
0620 06:02:16.989198 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:16.989214 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:16.989250 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:02:16.989950 (+ 700us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:16.989970 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:16.990397 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:16.990743 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:16.991182 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:16.992132 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:16.993120 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:16.993595 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:16.993767 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:16.994760 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:16.994765 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:16.994778 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:16.995179 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:16.995238 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:16.995257 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:16.995371 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:16.995526 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 06:02:16.995813 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":132,"threads_started":1}
I20260620 06:02:16.997979 8735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:16.998832 8735 log.cc:824] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:16.999492 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:02:17.001264 8735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c: No bootstrap required, opened a new log
I20260620 06:02:17.005213 8735 raft_consensus.cc:348] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 46103 } }
I20260620 06:02:17.005471 8735 raft_consensus.cc:374] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:17.005554 8735 raft_consensus.cc:729] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 372d40ef17e94e2fb4a15ad01c256a6c, State: Initialized, Role: FOLLOWER
I20260620 06:02:17.006032 8735 consensus_queue.cc:260] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 46103 } }
I20260620 06:02:17.006242 8735 raft_consensus.cc:388] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:17.006357 8735 raft_consensus.cc:482] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:17.006480 8735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:17.007522 8735 raft_consensus.cc:504] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 46103 } }
I20260620 06:02:17.007880 8735 leader_election.cc:302] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 372d40ef17e94e2fb4a15ad01c256a6c; no voters:
I20260620 06:02:17.008236 8735 leader_election.cc:288] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:17.008359 8740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:17.008612 8740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [term 1 LEADER]: Becoming Leader. State: Replica: 372d40ef17e94e2fb4a15ad01c256a6c, State: Running, Role: LEADER
I20260620 06:02:17.008942 8740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 46103 } }
I20260620 06:02:17.009553 8735 sys_catalog.cc:564] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:17.010679 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 372d40ef17e94e2fb4a15ad01c256a6c. Latest consensus state: current_term: 1 leader_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 46103 } } }
I20260620 06:02:17.010807 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.011072 8741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "372d40ef17e94e2fb4a15ad01c256a6c" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 46103 } } }
I20260620 06:02:17.011173 8741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.011232 8749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:17.014019 8749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:17.019598 8749 catalog_manager.cc:1347] Generated new cluster ID: ae6ec1aed6a04d76bc4fb5aac4260c5d
I20260620 06:02:17.019675 8749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:17.037488 8749 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:17.038457 8749 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:17.050763 8749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 372d40ef17e94e2fb4a15ad01c256a6c: Generated new TSK 0
I20260620 06:02:17.051501 8749 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:02:17.086658 8765 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 06:02:17.086788 8765 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:02:17.087543 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:17.069541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36466 (local address 127.3.115.62:46103)
0620 06:02:17.069672 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:17.069677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:17.069918 (+ 241us) server_negotiation.cc:407] Connection header received
0620 06:02:17.070177 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:17.070180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:17.070242 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:17.070350 (+ 108us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:02:17.070604 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.071202 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.072112 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.072351 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.072462 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.072866 (+ 404us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:02:17.087100 (+ 14234us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:02:17.087390 (+ 290us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 06:02:17.087612 8738 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 06:02:17.088791 8764 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:17.069414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:46103 (local address 127.0.0.1:36466)
0620 06:02:17.069827 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:17.069839 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:17.069992 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:17.070365 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:17.070368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:17.070377 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:02:17.070494 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:17.070499 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.071343 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.071349 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:17.071986 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:17.071993 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.072470 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.072474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:17.072568 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.072611 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:02:17.087342 (+ 14731us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 06:02:17.087546 (+ 204us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.115.62:46103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":112,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.115.62:46103: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37401/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 06:02:17.089466 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8668
2026-06-20T06:02:17Z chronyd exiting
I20260620 06:02:17.126930 3532 test_util.cc:175] -----------------------------------------------
I20260620 06:02:17.127022 3532 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781935311537307-3532-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (328 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 06:02:17.128052 3532 mini_oidc.cc:152] Starting JWKS server
I20260620 06:02:17.129001 3532 webserver.cc:460] Webserver started at https://127.0.0.1:39659/ using document root <none> and password file <none>
I20260620 06:02:17.129112 3532 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:02:17.129431 3532 webserver.cc:460] Webserver started at http://127.0.0.1:34397/ using document root <none> and password file <none>
2026-06-20T06:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:17Z Disabled control of system clock
I20260620 06:02:17.138370 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:44223
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:44223
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39659/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/testchainca.cert with env {}
W20260620 06:02:17.248803 8771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:17.249147 8771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:17.249238 8771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:17.253383 8771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:17.253465 8771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:17.253492 8771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:17.253532 8771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:17.253566 8771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:02:17.253603 8771 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:02:17.253631 8771 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39659/jwks.json
I20260620 06:02:17.259002 8771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:44223
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:44223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39659/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:17.260099 8771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:17.261166 8771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:17.269711 8777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:17.269690 8779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:17.269789 8776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:17.270143 8771 server_base.cc:1034] running on GCE node
I20260620 06:02:17.270763 8771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:17.271895 8771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:17.273065 8771 hybrid_clock.cc:648] HybridClock initialized: now 1781935337273038 us; error 42 us; skew 500 ppm
I20260620 06:02:17.275027 8771 webserver.cc:460] Webserver started at http://127.3.115.62:36827/ using document root <none> and password file <none>
I20260620 06:02:17.275612 8771 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:17.275676 8771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:17.275929 8771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:17.278017 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "89d10e3439c74680baaca56286fd3f33"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "7f4b0f325bfabe4d76fc93d2cb9d4e98"
server_key_iv: "369c0ef882be3651aa4881c814fb4bbf"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.278590 8771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "89d10e3439c74680baaca56286fd3f33"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "7f4b0f325bfabe4d76fc93d2cb9d4e98"
server_key_iv: "369c0ef882be3651aa4881c814fb4bbf"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.283254 8771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:17.287364 8785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:17.289280 8771 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:02:17.289424 8771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "89d10e3439c74680baaca56286fd3f33"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "7f4b0f325bfabe4d76fc93d2cb9d4e98"
server_key_iv: "369c0ef882be3651aa4881c814fb4bbf"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.289597 8771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:17.317283 8771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:17.318041 8771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:17.318252 8771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:17.325132 8771 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:44223
I20260620 06:02:17.325197 8837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:44223 every 8 connection(s)
I20260620 06:02:17.326126 8771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:17.329295 8838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:17.334525 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8771
I20260620 06:02:17.334633 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:17.334921 3532 external_mini_cluster.cc:1428] Setting key 5561251871d094675cd6b9f8e1b764b2
I20260620 06:02:17.335275 8838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33: Bootstrap starting.
I20260620 06:02:17.337738 8838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:17.338471 8838 log.cc:824] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:17.340478 8838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33: No bootstrap required, opened a new log
I20260620 06:02:17.344712 8838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d10e3439c74680baaca56286fd3f33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 44223 } }
I20260620 06:02:17.344933 8838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:17.345018 8838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d10e3439c74680baaca56286fd3f33, State: Initialized, Role: FOLLOWER
I20260620 06:02:17.344954 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:17.336445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.3.115.62:44223)
0620 06:02:17.337172 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:17.337187 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:17.337225 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:02:17.337934 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:17.337953 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:17.338356 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:17.338714 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:17.339171 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.340189 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.341512 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.341970 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.342092 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.342966 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:17.342971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:17.342984 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:17.343414 (+ 430us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:17.343470 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:17.343489 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:17.343574 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:17.343760 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 06:02:17.344060 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":207,"threads_started":1}
I20260620 06:02:17.345541 8838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d10e3439c74680baaca56286fd3f33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 44223 } }
I20260620 06:02:17.345700 8838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:17.345777 8838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:17.345898 8838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:17.347008 8838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d10e3439c74680baaca56286fd3f33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 44223 } }
I20260620 06:02:17.347354 8838 leader_election.cc:302] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89d10e3439c74680baaca56286fd3f33; no voters:
I20260620 06:02:17.347740 8838 leader_election.cc:288] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:17.347831 8843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:17.348104 8843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [term 1 LEADER]: Becoming Leader. State: Replica: 89d10e3439c74680baaca56286fd3f33, State: Running, Role: LEADER
I20260620 06:02:17.348153 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:02:17.348409 8843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d10e3439c74680baaca56286fd3f33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 44223 } }
I20260620 06:02:17.349047 8838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:17.350106 8844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89d10e3439c74680baaca56286fd3f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d10e3439c74680baaca56286fd3f33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 44223 } } }
I20260620 06:02:17.350230 8844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.350585 8845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89d10e3439c74680baaca56286fd3f33. Latest consensus state: current_term: 1 leader_uuid: "89d10e3439c74680baaca56286fd3f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d10e3439c74680baaca56286fd3f33" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 44223 } } }
I20260620 06:02:17.350667 8845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.350943 8852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:17.354432 8852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:17.360616 8852 catalog_manager.cc:1347] Generated new cluster ID: c0f1e679c7ce42f9b7f2bcb8bcdf341e
I20260620 06:02:17.360713 8852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:17.375042 8852 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:17.376044 8852 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:17.381285 8852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89d10e3439c74680baaca56286fd3f33: Generated new TSK 0
I20260620 06:02:17.381994 8852 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 06:02:17.414848 8868 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 06:02:17.414952 8868 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 06:02:17.415618 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:17.391929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.3.115.62:44223)
0620 06:02:17.392067 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:17.392072 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:17.392291 (+ 219us) server_negotiation.cc:407] Connection header received
0620 06:02:17.392413 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:17.392416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:17.392462 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:17.392556 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:02:17.392819 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.393351 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.394630 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.394882 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.394975 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.395316 (+ 341us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:02:17.415231 (+ 19915us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:02:17.415484 (+ 253us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43276: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 06:02:17.415711 8841 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43276: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 06:02:17.415819 8867 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:17.391862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:44223 (local address 127.0.0.1:43276)
0620 06:02:17.392169 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:17.392184 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:17.392302 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:17.392580 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:17.392583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:17.392592 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:02:17.392723 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:17.392730 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.393478 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.393482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:17.394513 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:17.394522 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.395002 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.395006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:17.395124 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.395154 (+ 30us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:02:17.415443 (+ 20289us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 06:02:17.415641 (+ 198us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.115.62:44223: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39659/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":127,"threads_started":1}
I20260620 06:02:17.416627 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8771
2026-06-20T06:02:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 06:02:17.455850 3532 mini_oidc.cc:152] Starting JWKS server
I20260620 06:02:17.456769 3532 webserver.cc:460] Webserver started at https://127.0.0.1:45077/ using document root <none> and password file <none>
I20260620 06:02:17.456887 3532 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 06:02:17.457177 3532 webserver.cc:460] Webserver started at http://127.0.0.1:38419/ using document root <none> and password file <none>
2026-06-20T06:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:17Z Disabled control of system clock
I20260620 06:02:17.466466 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45261
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:45261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45077/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:02:17.576560 8874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:17.576979 8874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:17.577059 8874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:17.581068 8874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:17.581166 8874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:17.581194 8874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:17.581235 8874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:17.581269 8874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:02:17.581305 8874 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:02:17.581334 8874 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45077/jwks.json
I20260620 06:02:17.586233 8874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:45261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45077/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:17.587441 8874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:17.588953 8874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:17.597330 8879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:17.597307 8880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:17.597307 8882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:17.597800 8874 server_base.cc:1034] running on GCE node
I20260620 06:02:17.598357 8874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:17.599474 8874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:17.600708 8874 hybrid_clock.cc:648] HybridClock initialized: now 1781935337600682 us; error 34 us; skew 500 ppm
I20260620 06:02:17.602687 8874 webserver.cc:460] Webserver started at http://127.3.115.62:38059/ using document root <none> and password file <none>
I20260620 06:02:17.603291 8874 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:17.603389 8874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:17.603626 8874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:17.605796 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "d99e31cc91a34e469ac3705b3c8a5ce8"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "aed45d59aa659226547a27695cfd78d5"
server_key_iv: "e46883da0f5cf313c9db6d39ff64f15b"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.606397 8874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "d99e31cc91a34e469ac3705b3c8a5ce8"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "aed45d59aa659226547a27695cfd78d5"
server_key_iv: "e46883da0f5cf313c9db6d39ff64f15b"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.610179 8874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:02:17.612608 8888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:17.613835 8874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:17.613976 8874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "d99e31cc91a34e469ac3705b3c8a5ce8"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "aed45d59aa659226547a27695cfd78d5"
server_key_iv: "e46883da0f5cf313c9db6d39ff64f15b"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.614100 8874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:17.630817 8874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:17.631572 8874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:17.631776 8874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:17.638701 8874 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:45261
I20260620 06:02:17.638720 8940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:45261 every 8 connection(s)
I20260620 06:02:17.639657 8874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:17.641827 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8874
I20260620 06:02:17.641965 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:17.642228 3532 external_mini_cluster.cc:1428] Setting key 84fe7773804fb80c7e500d4376d752ff
I20260620 06:02:17.643175 8941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:17.649520 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8: Bootstrap starting.
I20260620 06:02:17.651510 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:17.643603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44060 (local address 127.3.115.62:45261)
0620 06:02:17.644245 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:17.644261 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:17.644297 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:02:17.645020 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:17.645038 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:17.645433 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:17.645795 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:17.646227 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.647190 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.648112 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.648596 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.648832 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.649770 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:17.649775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:17.649789 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:17.650187 (+ 398us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:17.650242 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:17.650262 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:17.650374 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:17.650499 (+ 125us) server_negotiation.cc:299] Negotiation successful
0620 06:02:17.650774 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":123,"threads_started":1}
I20260620 06:02:17.652047 8941 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:17.652951 8941 log.cc:824] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:17.654703 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:02:17.655237 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8: No bootstrap required, opened a new log
I20260620 06:02:17.658938 8941 raft_consensus.cc:348] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45261 } }
I20260620 06:02:17.659205 8941 raft_consensus.cc:374] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:17.659293 8941 raft_consensus.cc:729] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99e31cc91a34e469ac3705b3c8a5ce8, State: Initialized, Role: FOLLOWER
I20260620 06:02:17.659762 8941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45261 } }
I20260620 06:02:17.659909 8941 raft_consensus.cc:388] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:17.659993 8941 raft_consensus.cc:482] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:17.660101 8941 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:17.661240 8941 raft_consensus.cc:504] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45261 } }
I20260620 06:02:17.661615 8941 leader_election.cc:302] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d99e31cc91a34e469ac3705b3c8a5ce8; no voters:
I20260620 06:02:17.661960 8941 leader_election.cc:288] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:17.662082 8946 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:17.662333 8946 raft_consensus.cc:686] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [term 1 LEADER]: Becoming Leader. State: Replica: d99e31cc91a34e469ac3705b3c8a5ce8, State: Running, Role: LEADER
I20260620 06:02:17.662726 8946 consensus_queue.cc:237] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45261 } }
I20260620 06:02:17.663364 8941 sys_catalog.cc:564] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:17.664479 8947 sys_catalog.cc:455] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45261 } } }
I20260620 06:02:17.664609 8947 sys_catalog.cc:458] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.664956 8948 sys_catalog.cc:455] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d99e31cc91a34e469ac3705b3c8a5ce8. Latest consensus state: current_term: 1 leader_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d99e31cc91a34e469ac3705b3c8a5ce8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45261 } } }
I20260620 06:02:17.665060 8948 sys_catalog.cc:458] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.665174 8955 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:17.668931 8955 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:17.676148 8955 catalog_manager.cc:1347] Generated new cluster ID: 443abc142ac045c7ac6e197308ae2d96
I20260620 06:02:17.676242 8955 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:17.687271 8955 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:17.688606 8955 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:17.693379 8955 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d99e31cc91a34e469ac3705b3c8a5ce8: Generated new TSK 0
I20260620 06:02:17.694206 8955 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:17.707412 8944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:17.702573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44076 (local address 127.3.115.62:45261)
0620 06:02:17.702699 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:17.702703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:17.702817 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:02:17.703044 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:17.703048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:17.703103 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:17.703222 (+ 119us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:02:17.703469 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.704032 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.705004 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.705228 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.705340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.705642 (+ 302us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:02:17.707044 (+ 1402us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45077/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:02:17.707266 (+ 222us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45077/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":55}
W20260620 06:02:17.707486 8944 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44076: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45077/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 06:02:17.709410 8970 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:17.702404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45261 (local address 127.0.0.1:44076)
0620 06:02:17.702714 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:17.702731 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:17.702879 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:17.703231 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:17.703235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:17.703243 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:02:17.703359 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:17.703365 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.704161 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.704166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:17.704885 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:17.704893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.705350 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.705354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:17.705441 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.705465 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:02:17.707212 (+ 1747us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45077/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 06:02:17.709186 (+ 1974us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.115.62:45261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45077/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1757,"thread_start_us":104,"threads_started":1}
I20260620 06:02:17.710355 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8874
E20260620 06:02:17.718307 8971 webserver.cc:533] Webserver: sslize failed:
E20260620 06:02:17.718374 8971 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T06:02:17Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (293 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T06:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:17Z Disabled control of system clock
I20260620 06:02:17.758054 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45609
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:45609
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/testchainca.cert with env {}
W20260620 06:02:17.872823 8975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:17.873113 8975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:17.873155 8975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:17.877748 8975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:17.877808 8975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:17.877825 8975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:17.877841 8975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:17.877857 8975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 06:02:17.877873 8975 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 06:02:17.877887 8975 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 06:02:17.883204 8975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:45609
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:17.884234 8975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:17.885283 8975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:17.893546 8983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:17.893563 8981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:17.893589 8980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:17.893805 8975 server_base.cc:1034] running on GCE node
I20260620 06:02:17.894450 8975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:17.895588 8975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:17.896728 8975 hybrid_clock.cc:648] HybridClock initialized: now 1781935337896717 us; error 35 us; skew 500 ppm
I20260620 06:02:17.898689 8975 webserver.cc:460] Webserver started at http://127.3.115.62:35193/ using document root <none> and password file <none>
I20260620 06:02:17.899302 8975 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:17.899402 8975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:17.899641 8975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:17.901857 8975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "68f7248ccb7f4bb5b04c9b4c64243126"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "2482c685a8413ae19ea15fbb402e2e5c"
server_key_iv: "13a0e52a4e6ef10ed27bc073d98b6440"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.902482 8975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "68f7248ccb7f4bb5b04c9b4c64243126"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "2482c685a8413ae19ea15fbb402e2e5c"
server_key_iv: "13a0e52a4e6ef10ed27bc073d98b6440"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.906332 8975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:17.908846 8989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:17.910104 8975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:17.910252 8975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "68f7248ccb7f4bb5b04c9b4c64243126"
format_stamp: "Formatted at 2026-06-20 06:02:17 on dist-test-slave-4h14"
server_key: "2482c685a8413ae19ea15fbb402e2e5c"
server_key_iv: "13a0e52a4e6ef10ed27bc073d98b6440"
server_key_version: "encryptionkey@0"
I20260620 06:02:17.910370 8975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:17.931696 8975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:17.932456 8975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:17.932687 8975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:17.939568 8975 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:45609
I20260620 06:02:17.939579 9041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:45609 every 8 connection(s)
I20260620 06:02:17.940574 8975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:17.943985 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 8975
I20260620 06:02:17.943997 9042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:17.944183 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:17.944479 3532 external_mini_cluster.cc:1428] Setting key 0ea8ecaf826b10cbb48b75916a040476
I20260620 06:02:17.951156 9042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126: Bootstrap starting.
I20260620 06:02:17.953732 9042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:17.953928 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:17.945936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.3.115.62:45609)
0620 06:02:17.946686 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:17.946703 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:17.946739 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:02:17.947401 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:17.947418 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:17.947825 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:17.948105 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:17.948475 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.949498 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.950672 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:17.951120 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:17.951235 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:17.952117 (+ 882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:17.952123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:17.952140 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:17.952516 (+ 376us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:17.952569 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:17.952588 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:17.952740 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:17.952859 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 06:02:17.953151 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":128,"threads_started":1}
I20260620 06:02:17.954532 9042 log.cc:824] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:17.956475 9042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126: No bootstrap required, opened a new log
I20260620 06:02:17.956694 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:02:17.959894 9042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45609 } }
I20260620 06:02:17.960147 9042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:17.960230 9042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f7248ccb7f4bb5b04c9b4c64243126, State: Initialized, Role: FOLLOWER
I20260620 06:02:17.960704 9042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45609 } }
I20260620 06:02:17.960867 9042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:17.960937 9042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:17.961035 9042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:17.961933 9042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45609 } }
I20260620 06:02:17.962275 9042 leader_election.cc:302] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68f7248ccb7f4bb5b04c9b4c64243126; no voters:
I20260620 06:02:17.962621 9042 leader_election.cc:288] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:17.962738 9047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:17.962929 9047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [term 1 LEADER]: Becoming Leader. State: Replica: 68f7248ccb7f4bb5b04c9b4c64243126, State: Running, Role: LEADER
I20260620 06:02:17.963250 9047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45609 } }
I20260620 06:02:17.963922 9042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:17.964939 9048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45609 } } }
I20260620 06:02:17.965075 9048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.965366 9049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68f7248ccb7f4bb5b04c9b4c64243126. Latest consensus state: current_term: 1 leader_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68f7248ccb7f4bb5b04c9b4c64243126" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45609 } } }
I20260620 06:02:17.965456 9049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:17.965518 9056 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:17.968701 9056 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:17.974244 9056 catalog_manager.cc:1347] Generated new cluster ID: 0688ae17a0b94e8d8e6d1dbaa1c25f77
I20260620 06:02:17.974314 9056 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:17.991736 9056 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:17.992858 9056 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:17.998678 9056 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68f7248ccb7f4bb5b04c9b4c64243126: Generated new TSK 0
I20260620 06:02:17.999374 9056 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:18.087420 9045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:18.010939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42322 (local address 127.3.115.62:45609)
0620 06:02:18.011068 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:18.011075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:18.011260 (+ 185us) server_negotiation.cc:407] Connection header received
0620 06:02:18.011464 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:18.011466 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:18.011512 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:18.011628 (+ 116us) server_negotiation.cc:226] Negotiated authn=JWT
0620 06:02:18.011872 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:18.012413 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:18.013368 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:18.013614 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:18.013730 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:18.014088 (+ 358us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 06:02:18.087029 (+ 72941us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:02:18.087301 (+ 272us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42322: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:02:18.087689 9045 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42322: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 06:02:18.087710 9071 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:18.010849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45609 (local address 127.0.0.1:42322)
0620 06:02:18.011172 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:18.011185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:18.011314 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:18.011630 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:18.011633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:18.011642 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0620 06:02:18.011757 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:18.011770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:18.012529 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:18.012533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:18.013244 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:18.013252 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:18.013739 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:18.013743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:18.013832 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:18.013864 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 06:02:18.087284 (+ 73420us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 06:02:18.087508 (+ 224us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.115.62:45609: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":109,"threads_started":1}
I20260620 06:02:18.088517 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 8975
2026-06-20T06:02:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (358 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (123 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (135 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:18 dist-test-slave-4h14 krb5kdc[9083](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:18 dist-test-slave-4h14 krb5kdc[9083](info): set up 2 sockets
Jun 20 06:02:18 dist-test-slave-4h14 krb5kdc[9083](info): commencing operation
krb5kdc: starting...
W20260620 06:02:19.381567 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:19 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:19.501060 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43891
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:19.612421 9096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:19.612784 9096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:19.612872 9096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:19.616936 9096 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:19.617029 9096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:19.617069 9096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:19.617127 9096 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:19.617167 9096 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:19.622457 9096 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:19.623592 9096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:19.624711 9096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:19.633070 9104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:19.633070 9101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:19.633168 9102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:19.633497 9096 server_base.cc:1034] running on GCE node
I20260620 06:02:19.634093 9096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:19.635241 9096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:19.636412 9096 hybrid_clock.cc:648] HybridClock initialized: now 1781935339636382 us; error 44 us; skew 500 ppm
Jun 20 06:02:19 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:19.642354 9096 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:19.643293 9096 webserver.cc:460] Webserver started at http://127.3.115.62:40947/ using document root <none> and password file <none>
I20260620 06:02:19.643874 9096 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:19.643978 9096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:19.644217 9096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:19.646481 9096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "b68271a456464f24b1b6508c750fdd55"
format_stamp: "Formatted at 2026-06-20 06:02:19 on dist-test-slave-4h14"
server_key: "2f65409d7b8e30b685096b02606b9c2d"
server_key_iv: "7020ec4434cf893f9278177cbf1ab237"
server_key_version: "encryptionkey@0"
I20260620 06:02:19.647114 9096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "b68271a456464f24b1b6508c750fdd55"
format_stamp: "Formatted at 2026-06-20 06:02:19 on dist-test-slave-4h14"
server_key: "2f65409d7b8e30b685096b02606b9c2d"
server_key_iv: "7020ec4434cf893f9278177cbf1ab237"
server_key_version: "encryptionkey@0"
I20260620 06:02:19.651221 9096 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 06:02:19.653891 9111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:19.655191 9096 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:19.655345 9096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "b68271a456464f24b1b6508c750fdd55"
format_stamp: "Formatted at 2026-06-20 06:02:19 on dist-test-slave-4h14"
server_key: "2f65409d7b8e30b685096b02606b9c2d"
server_key_iv: "7020ec4434cf893f9278177cbf1ab237"
server_key_version: "encryptionkey@0"
I20260620 06:02:19.655465 9096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:19.680483 9096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:19.683542 9096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:19.683766 9096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:19.690907 9096 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:43891
I20260620 06:02:19.690984 9163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:43891 every 8 connection(s)
I20260620 06:02:19.691906 9096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:19.694787 9164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:19.697469 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9096
I20260620 06:02:19.697615 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:19.697881 3532 external_mini_cluster.cc:1428] Setting key 054f6ab751a41a9caf2341284a41b607
I20260620 06:02:19.700876 9164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55: Bootstrap starting.
I20260620 06:02:19.703075 9164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:19.703848 9164 log.cc:824] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:19.705940 9164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55: No bootstrap required, opened a new log
Jun 20 06:02:19 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:19.709712 9164 raft_consensus.cc:348] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68271a456464f24b1b6508c750fdd55" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43891 } }
I20260620 06:02:19.709934 9164 raft_consensus.cc:374] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:19.709969 9164 raft_consensus.cc:729] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b68271a456464f24b1b6508c750fdd55, State: Initialized, Role: FOLLOWER
I20260620 06:02:19.710492 9164 consensus_queue.cc:260] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68271a456464f24b1b6508c750fdd55" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43891 } }
I20260620 06:02:19.710628 9164 raft_consensus.cc:388] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:19.710716 9164 raft_consensus.cc:482] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:19.710829 9164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:19.711736 9164 raft_consensus.cc:504] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68271a456464f24b1b6508c750fdd55" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43891 } }
I20260620 06:02:19.712123 9164 leader_election.cc:302] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b68271a456464f24b1b6508c750fdd55; no voters:
I20260620 06:02:19.712493 9164 leader_election.cc:288] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:19.712576 9169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:19.712786 9169 raft_consensus.cc:686] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [term 1 LEADER]: Becoming Leader. State: Replica: b68271a456464f24b1b6508c750fdd55, State: Running, Role: LEADER
I20260620 06:02:19.713084 9169 consensus_queue.cc:237] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68271a456464f24b1b6508c750fdd55" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43891 } }
I20260620 06:02:19.713374 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:19.699354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.3.115.62:43891)
0620 06:02:19.699811 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:19.699826 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:19.699861 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:02:19.700477 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:19.700495 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:19.700913 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:19.701295 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:19.702073 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:19.703060 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:19.703957 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:19.704491 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:19.704694 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:19.707469 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:19.707474 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:19.707489 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:19.707517 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:19.709955 (+ 2438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:19.710444 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:19.710450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:19.710463 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:19.710581 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:19.710965 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:19.710968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:19.710970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:19.711416 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:19.711627 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:19.711920 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 06:02:19.712240 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260620 06:02:19.713835 9164 sys_catalog.cc:564] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:19.714260 9169 sys_catalog.cc:455] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b68271a456464f24b1b6508c750fdd55. Latest consensus state: current_term: 1 leader_uuid: "b68271a456464f24b1b6508c750fdd55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68271a456464f24b1b6508c750fdd55" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43891 } } }
I20260620 06:02:19.714370 9169 sys_catalog.cc:458] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:19.714692 9170 sys_catalog.cc:455] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b68271a456464f24b1b6508c750fdd55" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b68271a456464f24b1b6508c750fdd55" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43891 } } }
I20260620 06:02:19.714779 9170 sys_catalog.cc:458] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55 [sys.catalog]: This master's current role is: LEADER
W20260620 06:02:19.718278 9184 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:02:19.718371 9184 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:02:19.718436 9173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:19.719099 9173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:19.724586 9173 catalog_manager.cc:1347] Generated new cluster ID: 3602aaa5048346ada9b1ad59baa418bb
I20260620 06:02:19.724689 9173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:19.736231 9173 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:19.737562 9173 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:19.748389 9173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b68271a456464f24b1b6508c750fdd55: Generated new TSK 0
I20260620 06:02:19.749178 9173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:19.783622 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43891
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:19.927714 9190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:19.928010 9190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:19.928056 9190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:19.932060 9190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:19.932119 9190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:19.932178 9190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:19.937444 9190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:19.938473 9190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:19.939540 9190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:19.948997 9198 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:19.949019 9195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:19.948997 9196 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:19.949323 9190 server_base.cc:1034] running on GCE node
I20260620 06:02:19.949719 9190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:19.950424 9190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:19.951632 9190 hybrid_clock.cc:648] HybridClock initialized: now 1781935339951607 us; error 33 us; skew 500 ppm
Jun 20 06:02:19 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:19.954582 9190 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:19.955485 9190 webserver.cc:460] Webserver started at http://127.3.115.1:41057/ using document root <none> and password file <none>
I20260620 06:02:19.956037 9190 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:19.956125 9190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:19.956359 9190 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:19.958478 9190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "c2961bb8c2a24a579f6b17848cf2837c"
format_stamp: "Formatted at 2026-06-20 06:02:19 on dist-test-slave-4h14"
server_key: "476d5ba07998b17afcc3b7f47a01f417"
server_key_iv: "58d3d09adf7f5af3bba2f5e1c10f5f2d"
server_key_version: "encryptionkey@0"
I20260620 06:02:19.959048 9190 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "c2961bb8c2a24a579f6b17848cf2837c"
format_stamp: "Formatted at 2026-06-20 06:02:19 on dist-test-slave-4h14"
server_key: "476d5ba07998b17afcc3b7f47a01f417"
server_key_iv: "58d3d09adf7f5af3bba2f5e1c10f5f2d"
server_key_version: "encryptionkey@0"
I20260620 06:02:19.962946 9190 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:19.965633 9205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:19.966908 9190 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:19.967048 9190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "c2961bb8c2a24a579f6b17848cf2837c"
format_stamp: "Formatted at 2026-06-20 06:02:19 on dist-test-slave-4h14"
server_key: "476d5ba07998b17afcc3b7f47a01f417"
server_key_iv: "58d3d09adf7f5af3bba2f5e1c10f5f2d"
server_key_version: "encryptionkey@0"
I20260620 06:02:19.967167 9190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:19.984177 9190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:19.987257 9190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:19.987473 9190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:19.988085 9190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:19.988962 9190 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:19.989033 9190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:19.989102 9190 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:19.989151 9190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:19.997865 9190 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:40613
I20260620 06:02:19.997895 9318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:40613 every 8 connection(s)
I20260620 06:02:19.998745 9190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:20.001102 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9190
I20260620 06:02:20.001237 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:20.001600 3532 external_mini_cluster.cc:1428] Setting key 6d47718a53b29b50d6e99dde502bde3d
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935339, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:20.015874 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.000624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:52481 (local address 127.3.115.62:43891)
0620 06:02:20.000836 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.000841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.001849 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 06:02:20.003677 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.003679 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.003721 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.003787 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.005699 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.006560 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.008014 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.008266 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.008361 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.011526 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:20.011533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:20.011541 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:20.011577 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:20.013288 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.013968 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.013971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.013973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.014050 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.014628 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.014631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.014633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.014825 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:20.014938 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:20.015560 (+ 622us) server_negotiation.cc:299] Negotiation successful
0620 06:02:20.015694 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260620 06:02:20.016829 9321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.001077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43891 (local address 127.3.115.1:52481)
0620 06:02:20.001654 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:20.001721 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:20.003440 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:20.004021 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:20.004034 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:20.004679 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:20.005501 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:20.005523 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.006710 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.006714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:20.007825 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:20.007835 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.008449 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.008456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:20.008757 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.009475 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:20.009507 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:20.011336 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:20.013434 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:20.013442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:20.013461 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:20.013826 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:20.014157 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:20.014160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:20.014162 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:20.014487 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:20.014934 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:20.014942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:20.015310 (+ 368us) client_negotiation.cc:769] Sending connection context
0620 06:02:20.015499 (+ 189us) client_negotiation.cc:240] Negotiation successful
0620 06:02:20.015844 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":166,"threads_started":1}
I20260620 06:02:20.018180 9319 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43891
I20260620 06:02:20.018471 9319 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:20.018999 9319 heartbeater.cc:511] Master 127.3.115.62:43891 requested a full tablet report, sending...
I20260620 06:02:20.020815 9128 ts_manager.cc:194] Registered new tserver with Master: c2961bb8c2a24a579f6b17848cf2837c (127.3.115.1:40613)
I20260620 06:02:20.022547 9128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:52481
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:20.031786 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43891
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:20.144019 9324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:20.144325 9324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:20.144369 9324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:20.148473 9324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:20.148545 9324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:20.148607 9324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:20.154497 9324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:20.155543 9324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:20.156600 9324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:20.166798 9332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:20.166956 9324 server_base.cc:1034] running on GCE node
W20260620 06:02:20.166805 9330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:20.166806 9329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:20.167586 9324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:20.168300 9324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:20.169483 9324 hybrid_clock.cc:648] HybridClock initialized: now 1781935340169466 us; error 31 us; skew 500 ppm
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.172572 9324 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:20.173528 9324 webserver.cc:460] Webserver started at http://127.3.115.2:43553/ using document root <none> and password file <none>
I20260620 06:02:20.174064 9324 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:20.174124 9324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:20.174300 9324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:20.176349 9324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "aa28058b04c34b458f6ba28dd377c6a9"
format_stamp: "Formatted at 2026-06-20 06:02:20 on dist-test-slave-4h14"
server_key: "d96419e5be76a7ae7a4d98aa2f5fb70d"
server_key_iv: "ddce42bdc888d5a5e9a22c8b23b67762"
server_key_version: "encryptionkey@0"
I20260620 06:02:20.176932 9324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "aa28058b04c34b458f6ba28dd377c6a9"
format_stamp: "Formatted at 2026-06-20 06:02:20 on dist-test-slave-4h14"
server_key: "d96419e5be76a7ae7a4d98aa2f5fb70d"
server_key_iv: "ddce42bdc888d5a5e9a22c8b23b67762"
server_key_version: "encryptionkey@0"
I20260620 06:02:20.180635 9324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:02:20.183151 9339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:20.184413 9324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:20.184557 9324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "aa28058b04c34b458f6ba28dd377c6a9"
format_stamp: "Formatted at 2026-06-20 06:02:20 on dist-test-slave-4h14"
server_key: "d96419e5be76a7ae7a4d98aa2f5fb70d"
server_key_iv: "ddce42bdc888d5a5e9a22c8b23b67762"
server_key_version: "encryptionkey@0"
I20260620 06:02:20.184747 9324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:20.198865 9324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:20.201793 9324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:20.201994 9324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:20.202605 9324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:20.203449 9324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:20.203495 9324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:20.203562 9324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:20.203591 9324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:20.212716 9324 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:36121
I20260620 06:02:20.212762 9452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:36121 every 8 connection(s)
I20260620 06:02:20.213663 9324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:20.218055 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9324
I20260620 06:02:20.218192 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:20.218485 3532 external_mini_cluster.cc:1428] Setting key f34e33cf945c8d845067b28005759d27
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:02:20.229007 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.215607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:38703 (local address 127.3.115.62:43891)
0620 06:02:20.215783 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.215792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.216800 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 06:02:20.218164 (+ 1364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.218168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.218232 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.218332 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.219636 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.220280 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.221658 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.221880 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.221960 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.224994 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:20.225001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:20.225008 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:20.225045 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:20.226719 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.227248 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.227252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.227254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.227344 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.227716 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.227721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.227726 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.227978 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:20.228091 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:20.228597 (+ 506us) server_negotiation.cc:299] Negotiation successful
0620 06:02:20.228782 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:02:20.229895 9455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.216106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43891 (local address 127.3.115.2:38703)
0620 06:02:20.216588 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:20.216633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:20.217956 (+ 1323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:20.218467 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:20.218474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:20.218917 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:20.219483 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:20.219496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.220409 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.220419 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:20.221512 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:20.221527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.222032 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.222037 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:20.222275 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.222915 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:20.222941 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:20.224810 (+ 1869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:20.226847 (+ 2037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:20.226852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:20.226864 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:20.227127 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:20.227444 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:20.227446 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:20.227448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:20.227581 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:20.228092 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:20.228096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:20.228336 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 06:02:20.228489 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 06:02:20.228867 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":163,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:02:20.231200 9453 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43891
I20260620 06:02:20.231410 9453 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:20.231849 9453 heartbeater.cc:511] Master 127.3.115.62:43891 requested a full tablet report, sending...
I20260620 06:02:20.232909 9128 ts_manager.cc:194] Registered new tserver with Master: aa28058b04c34b458f6ba28dd377c6a9 (127.3.115.2:36121)
I20260620 06:02:20.233481 9128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:38703
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:20.248791 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43891
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:20.361840 9458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:20.362124 9458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:20.362205 9458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:20.366567 9458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:20.366631 9458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:20.366740 9458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:20.372184 9458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:20.373304 9458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:20.374328 9458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:20.384070 9464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:20.384063 9458 server_base.cc:1034] running on GCE node
W20260620 06:02:20.383991 9463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:20.384012 9466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:20.384701 9458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:20.385402 9458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:20.386580 9458 hybrid_clock.cc:648] HybridClock initialized: now 1781935340386566 us; error 29 us; skew 500 ppm
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.389621 9458 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:20.390545 9458 webserver.cc:460] Webserver started at http://127.3.115.3:43257/ using document root <none> and password file <none>
I20260620 06:02:20.391134 9458 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:20.391228 9458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:20.391466 9458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:20.393590 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "fc1dac8bd6764aa2989cfa14544ace85"
format_stamp: "Formatted at 2026-06-20 06:02:20 on dist-test-slave-4h14"
server_key: "ddb4da0a9239a075e080cebc234f62df"
server_key_iv: "a922a098d5096fcc84bb501c5a8cbb1c"
server_key_version: "encryptionkey@0"
I20260620 06:02:20.394156 9458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "fc1dac8bd6764aa2989cfa14544ace85"
format_stamp: "Formatted at 2026-06-20 06:02:20 on dist-test-slave-4h14"
server_key: "ddb4da0a9239a075e080cebc234f62df"
server_key_iv: "a922a098d5096fcc84bb501c5a8cbb1c"
server_key_version: "encryptionkey@0"
I20260620 06:02:20.398051 9458 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:20.400749 9473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:20.402215 9458 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:20.402382 9458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "fc1dac8bd6764aa2989cfa14544ace85"
format_stamp: "Formatted at 2026-06-20 06:02:20 on dist-test-slave-4h14"
server_key: "ddb4da0a9239a075e080cebc234f62df"
server_key_iv: "a922a098d5096fcc84bb501c5a8cbb1c"
server_key_version: "encryptionkey@0"
I20260620 06:02:20.402525 9458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:20.412844 9458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:20.416345 9458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:20.416561 9458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:20.417243 9458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:20.418107 9458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:20.418179 9458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:20.418253 9458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:20.418298 9458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:20.427534 9458 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:45437
I20260620 06:02:20.427575 9586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:45437 every 8 connection(s)
I20260620 06:02:20.428495 9458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:20.435227 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9458
I20260620 06:02:20.435380 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:20.435691 3532 external_mini_cluster.cc:1428] Setting key f79ef020b8138a5fcaaae496096548f5
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:20.443589 9167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.430696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:36107 (local address 127.3.115.62:43891)
0620 06:02:20.430888 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.430891 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.431793 (+ 902us) server_negotiation.cc:407] Connection header received
0620 06:02:20.433322 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.433325 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.433384 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.433468 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.434742 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.435300 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.436835 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.437065 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.437183 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.439957 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:20.439961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:20.439963 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:20.439988 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:20.441419 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.441948 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.441953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.441955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.442040 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.442396 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.442401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.442403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.442595 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:20.442710 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:20.443288 (+ 578us) server_negotiation.cc:299] Negotiation successful
0620 06:02:20.443419 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260620 06:02:20.444502 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.431167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43891 (local address 127.3.115.3:36107)
0620 06:02:20.431632 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:20.431679 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:20.433056 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:20.433648 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:20.433657 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:20.434048 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:20.434580 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:20.434594 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.435440 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.435446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:20.436707 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:20.436719 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.437228 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.437232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:20.437440 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.438245 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:20.438262 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:20.439779 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:20.441545 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:20.441551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:20.441568 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:20.441841 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:20.442142 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:20.442145 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:20.442147 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:20.442298 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:20.442708 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:20.442715 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:20.443024 (+ 309us) client_negotiation.cc:769] Sending connection context
0620 06:02:20.443249 (+ 225us) client_negotiation.cc:240] Negotiation successful
0620 06:02:20.443608 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":121,"threads_started":1}
I20260620 06:02:20.445755 9587 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43891
I20260620 06:02:20.446038 9587 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:20.446568 9587 heartbeater.cc:511] Master 127.3.115.62:43891 requested a full tablet report, sending...
I20260620 06:02:20.447592 9128 ts_manager.cc:194] Registered new tserver with Master: fc1dac8bd6764aa2989cfa14544ace85 (127.3.115.3:45437)
I20260620 06:02:20.448173 9128 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:36107
I20260620 06:02:20.450135 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/test-admin.keytab.
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.466756 3532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 06:02:20.469671 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:20.468118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56136 (local address 127.3.115.1:40613)
0620 06:02:20.468424 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.468449 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.468473 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:02:20.468526 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.468530 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.468838 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.468995 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.469463 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56136: BlockingRecv error: recv got EOF from 127.0.0.1:56136 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":140,"threads_started":1}
W20260620 06:02:20.469762 3532 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.470862 3532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.471906 3532 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:02:20.481585 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.472874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56150 (local address 127.3.115.1:40613)
0620 06:02:20.473003 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.473007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.473035 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:02:20.473170 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.473172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.473211 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.473278 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.473806 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.474610 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.475424 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.475814 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.475902 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.478203 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:20.478212 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:20.478226 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:20.478250 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:20.479928 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.480315 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.480321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.480322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.480397 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.480635 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.480637 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.480638 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.480914 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:20.481065 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:20.481248 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 06:02:20.481409 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 06:02:20.485443 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:20.484423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56158 (local address 127.3.115.1:40613)
0620 06:02:20.484543 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.484547 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.484683 (+ 136us) server_negotiation.cc:407] Connection header received
0620 06:02:20.484821 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.484824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.484867 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.484948 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.485329 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56158: BlockingRecv error: recv got EOF from 127.0.0.1:56158 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 06:02:20.485584 3532 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.486541 3532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.487391 3532 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:02:20.495821 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.488248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56168 (local address 127.3.115.1:40613)
0620 06:02:20.488371 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.488375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.488469 (+ 94us) server_negotiation.cc:407] Connection header received
0620 06:02:20.488535 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.488537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.488571 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.488640 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.489235 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.489763 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.490549 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.490779 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.490865 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.493043 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:20.493046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:20.493048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:20.493068 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:20.494268 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.494618 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.494620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.494622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.494687 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.494975 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.494977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.494979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.495138 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:20.495238 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:20.495420 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:02:20.495655 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260620 06:02:20.498701 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:20.497740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56170 (local address 127.3.115.1:40613)
0620 06:02:20.497855 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.497861 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.497985 (+ 124us) server_negotiation.cc:407] Connection header received
0620 06:02:20.498069 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.498071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.498105 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.498161 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.498594 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56170: BlockingRecv error: recv got EOF from 127.0.0.1:56170 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:02:20.498733 3532 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781935311537307-3532-0/krb5kdc/krb5cc' of type FILE
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.499727 3532 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:20.500695 3532 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9083](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:02:20.508754 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:20.501500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56184 (local address 127.3.115.1:40613)
0620 06:02:20.501645 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:20.501649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:20.501701 (+ 52us) server_negotiation.cc:407] Connection header received
0620 06:02:20.501796 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:20.501798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:20.501829 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:20.501890 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:20.502323 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.502806 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.503507 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:20.503721 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:20.503795 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:20.505974 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:20.505977 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:20.505979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:20.505999 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:20.507342 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.507747 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.507749 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.507750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.507820 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:20.508031 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:20.508033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:20.508034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:20.508175 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:20.508267 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:20.508431 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 06:02:20.508533 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:02:20.509739 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9190
I20260620 06:02:20.516288 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9324
I20260620 06:02:20.521595 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9458
I20260620 06:02:20.526889 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9096
2026-06-20T06:02:20Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2182 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9607](info): set up 2 sockets
Jun 20 06:02:20 dist-test-slave-4h14 krb5kdc[9607](info): commencing operation
krb5kdc: starting...
W20260620 06:02:21.570165 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:21 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:21Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab.
Entry for principal oryx/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab.
I20260620 06:02:21.695785 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35659
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab
--principal=oryx/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:35659
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:21.807386 9620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:21.807761 9620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:21.807873 9620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:21.813222 9620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:21.813324 9620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:21.813362 9620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:21.813400 9620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:21.813436 9620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:21.818868 9620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:35659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab
--principal=oryx/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:21.820029 9620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:21.821197 9620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:21.829592 9625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:21.829613 9628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:21.829618 9626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:21.829991 9620 server_base.cc:1034] running on GCE node
I20260620 06:02:21.830559 9620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:21.831684 9620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:21.832841 9620 hybrid_clock.cc:648] HybridClock initialized: now 1781935341832810 us; error 44 us; skew 500 ppm
Jun 20 06:02:21 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:21.835642 9620 init.cc:377] Logged in from keytab as oryx/127.3.115.62@KRBTEST.COM (short username oryx)
I20260620 06:02:21.836553 9620 webserver.cc:460] Webserver started at http://127.3.115.62:33959/ using document root <none> and password file <none>
I20260620 06:02:21.837172 9620 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:21.837275 9620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:21.837510 9620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:21.839552 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "7804e9288f6f4e179d4d76ffb820f536"
format_stamp: "Formatted at 2026-06-20 06:02:21 on dist-test-slave-4h14"
server_key: "e337adde12f7d4cf367f280015e143e0"
server_key_iv: "89c7c2392efb43dafe01126e13e2f8fc"
server_key_version: "encryptionkey@0"
I20260620 06:02:21.840126 9620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "7804e9288f6f4e179d4d76ffb820f536"
format_stamp: "Formatted at 2026-06-20 06:02:21 on dist-test-slave-4h14"
server_key: "e337adde12f7d4cf367f280015e143e0"
server_key_iv: "89c7c2392efb43dafe01126e13e2f8fc"
server_key_version: "encryptionkey@0"
I20260620 06:02:21.843834 9620 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:02:21.846472 9635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:21.847734 9620 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:21.847872 9620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "7804e9288f6f4e179d4d76ffb820f536"
format_stamp: "Formatted at 2026-06-20 06:02:21 on dist-test-slave-4h14"
server_key: "e337adde12f7d4cf367f280015e143e0"
server_key_iv: "89c7c2392efb43dafe01126e13e2f8fc"
server_key_version: "encryptionkey@0"
I20260620 06:02:21.847983 9620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:21.859761 9620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:21.862831 9620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:21.863052 9620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:21.870281 9620 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:35659
I20260620 06:02:21.870733 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:35659 every 8 connection(s)
I20260620 06:02:21.871311 9620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:21.872066 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9620
I20260620 06:02:21.872198 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:21.872489 3532 external_mini_cluster.cc:1428] Setting key c91d87f438ddfee51c55022a3fcb69ca
I20260620 06:02:21.875355 9688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:21.880827 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Bootstrap starting.
Jun 20 06:02:21 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:21.883142 9688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:21.884053 9688 log.cc:824] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:21.886286 9688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: No bootstrap required, opened a new log
I20260620 06:02:21.886328 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:21.873817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.3.115.62:35659)
0620 06:02:21.874201 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:21.874216 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:21.874253 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:02:21.874917 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:21.874932 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:21.875261 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:21.875603 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:21.876396 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:21.877441 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:21.878291 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:21.878762 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:21.878926 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:21.881559 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:21.881564 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:21.881576 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:21.881603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:21.883387 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:21.883843 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:21.883848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:21.883853 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:21.883962 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:21.884257 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:21.884260 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:21.884261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:21.884796 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:21.884978 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:21.885295 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 06:02:21.885610 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":114,"threads_started":1}
I20260620 06:02:21.890092 9688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } }
I20260620 06:02:21.890332 9688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:21.890412 9688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7804e9288f6f4e179d4d76ffb820f536, State: Initialized, Role: FOLLOWER
I20260620 06:02:21.890829 9688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } }
I20260620 06:02:21.890981 9688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:21.891059 9688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:21.891165 9688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:21.892108 9688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } }
I20260620 06:02:21.892436 9688 leader_election.cc:302] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7804e9288f6f4e179d4d76ffb820f536; no voters:
I20260620 06:02:21.892814 9688 leader_election.cc:288] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:21.892939 9693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:21.893216 9693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 LEADER]: Becoming Leader. State: Replica: 7804e9288f6f4e179d4d76ffb820f536, State: Running, Role: LEADER
I20260620 06:02:21.893577 9693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } }
I20260620 06:02:21.894090 9688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:21.895172 9695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7804e9288f6f4e179d4d76ffb820f536. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } }
I20260620 06:02:21.895296 9695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:21.895639 9694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } }
I20260620 06:02:21.895776 9694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:21.895704 9702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:21.898864 9702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:21.904273 9702 catalog_manager.cc:1347] Generated new cluster ID: ab969c50d19740719686c797238742e6
I20260620 06:02:21.904358 9702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:21.933115 9702 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:21.934127 9702 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:21.942955 9702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Generated new TSK 0
I20260620 06:02:21.943702 9702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab.
Entry for principal oryx/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab.
I20260620 06:02:21.977320 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35659
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:22.091163 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:22.091423 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:22.091464 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:22.095836 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:22.095896 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:22.095999 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:22.101227 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:22.102336 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:22.103443 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:22.113015 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:22.113011 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.113255 9714 server_base.cc:1034] running on GCE node
W20260620 06:02:22.113399 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.113840 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:22.114575 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:22.115756 9714 hybrid_clock.cc:648] HybridClock initialized: now 1781935342115736 us; error 32 us; skew 500 ppm
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:22.118643 9714 init.cc:377] Logged in from keytab as oryx/127.3.115.1@KRBTEST.COM (short username oryx)
I20260620 06:02:22.119586 9714 webserver.cc:460] Webserver started at http://127.3.115.1:41865/ using document root <none> and password file <none>
I20260620 06:02:22.120151 9714 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:22.120211 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:22.120450 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:22.122532 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "22662410bfb74fa091248719a263f65f"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "fdbf09c6afee970d70bf7417d72e3897"
server_key_iv: "75cd20f71947a14db129d1b8374323c3"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.123098 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "22662410bfb74fa091248719a263f65f"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "fdbf09c6afee970d70bf7417d72e3897"
server_key_iv: "75cd20f71947a14db129d1b8374323c3"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.126865 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:02:22.129312 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.130568 9714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:22.130698 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "22662410bfb74fa091248719a263f65f"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "fdbf09c6afee970d70bf7417d72e3897"
server_key_iv: "75cd20f71947a14db129d1b8374323c3"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.130805 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:22.142342 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:22.145628 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:22.145840 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:22.146438 9714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:22.147301 9714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:22.147349 9714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.147419 9714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:22.147452 9714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.156617 9714 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:45025
I20260620 06:02:22.156636 9842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:45025 every 8 connection(s)
I20260620 06:02:22.157691 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:22.163810 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9714
I20260620 06:02:22.163955 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:22.164304 3532 external_mini_cluster.cc:1428] Setting key d79523ec85c4bd275a955e3dfd0412bd
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:22.172787 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.159532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:60539 (local address 127.3.115.62:35659)
0620 06:02:22.159681 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.159685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.160631 (+ 946us) server_negotiation.cc:407] Connection header received
0620 06:02:22.161976 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.161979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.162027 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.162104 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.163253 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.163788 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.165267 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.165659 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.165764 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.168902 (+ 3138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.168905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.168907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.168930 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.170434 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.171080 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.171083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.171084 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.171160 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.171568 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.171570 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.171571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.171744 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.171817 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.172470 (+ 653us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.172571 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:02:22.173949 9845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.160004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:60539)
0620 06:02:22.160469 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.160513 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.161773 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.162283 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.162289 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.162642 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.163116 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.163130 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.163916 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.163921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.165146 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.165163 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.165841 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.165845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.166041 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.166697 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.166722 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.168750 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.170551 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.170557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.170576 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.170969 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.171257 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.171262 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.171267 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.171468 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.171848 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.171859 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.172210 (+ 351us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.172428 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.172822 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":112,"threads_started":1}
WARNING: no policy specified for oryx/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:02:22.175305 9843 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35659
I20260620 06:02:22.175591 9843 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:22.176105 9843 heartbeater.cc:511] Master 127.3.115.62:35659 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:22.177749 9652 ts_manager.cc:194] Registered new tserver with Master: 22662410bfb74fa091248719a263f65f (127.3.115.1:45025)
I20260620 06:02:22.179296 9652 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:60539
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab.
Entry for principal oryx/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab.
I20260620 06:02:22.191473 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35659
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:22.315891 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:22.316169 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:22.316213 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:22.320246 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:22.320318 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:22.320380 9848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:22.325816 9848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:22.326869 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:22.327858 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:22.337741 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:22.337761 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:22.337761 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.337939 9848 server_base.cc:1034] running on GCE node
I20260620 06:02:22.338495 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:22.339107 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:22.340265 9848 hybrid_clock.cc:648] HybridClock initialized: now 1781935342340229 us; error 50 us; skew 500 ppm
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:22.343132 9848 init.cc:377] Logged in from keytab as oryx/127.3.115.2@KRBTEST.COM (short username oryx)
I20260620 06:02:22.344084 9848 webserver.cc:460] Webserver started at http://127.3.115.2:33617/ using document root <none> and password file <none>
I20260620 06:02:22.344720 9848 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:22.344830 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:22.345069 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:22.347301 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "d09430c33e5d42eebc0403da38b85de9"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "79163e7bed62ff95c164ad2e9e545906"
server_key_iv: "afb32b74cdc646a791b7983046ab5ce9"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.347878 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "d09430c33e5d42eebc0403da38b85de9"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "79163e7bed62ff95c164ad2e9e545906"
server_key_iv: "afb32b74cdc646a791b7983046ab5ce9"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.351859 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:22.354519 9863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.355911 9848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:02:22.356072 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "d09430c33e5d42eebc0403da38b85de9"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "79163e7bed62ff95c164ad2e9e545906"
server_key_iv: "afb32b74cdc646a791b7983046ab5ce9"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.356204 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:22.388441 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:22.392016 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:22.392246 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:22.392935 9848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:22.393787 9848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:22.393862 9848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.393923 9848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:22.393956 9848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.403445 9848 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:46583
I20260620 06:02:22.403462 9976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:46583 every 8 connection(s)
I20260620 06:02:22.404428 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:22.408293 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9848
I20260620 06:02:22.408416 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:22.408787 3532 external_mini_cluster.cc:1428] Setting key 533c1451c748d5bfeb4e8704b47e732c
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.3@KRBTEST.COM" created.
I20260620 06:02:22.420400 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.406398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:35065 (local address 127.3.115.62:35659)
0620 06:02:22.406561 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.406564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.407624 (+ 1060us) server_negotiation.cc:407] Connection header received
0620 06:02:22.409289 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.409291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.409371 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.409467 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.411004 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.411529 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.412784 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.413014 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.413420 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.416385 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.416389 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.416391 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.416417 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.418003 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.418623 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.418628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.418629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.418708 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.419078 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.419082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.419084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.419350 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.419449 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.420088 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.420228 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:02:22.421366 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.406825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:35065)
0620 06:02:22.407450 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.407497 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.409012 (+ 1515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.409684 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.409693 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.410332 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.410856 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.410870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.411669 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.411672 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.412586 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.412595 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.413456 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.413461 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.413679 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.414353 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.414371 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.416153 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.418127 (+ 1974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.418134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.418152 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.418511 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.418818 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.418821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.418823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.418978 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.419463 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.419472 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.419777 (+ 305us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.420050 (+ 273us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.420423 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":116,"threads_started":1}
I20260620 06:02:22.422684 9977 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35659
I20260620 06:02:22.422986 9977 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:22.423475 9977 heartbeater.cc:511] Master 127.3.115.62:35659 requested a full tablet report, sending...
I20260620 06:02:22.424558 9652 ts_manager.cc:194] Registered new tserver with Master: d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583)
I20260620 06:02:22.425475 9652 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:35065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab.
Entry for principal oryx/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab.
I20260620 06:02:22.437659 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35659
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:22.551477 9982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:22.551749 9982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:22.551815 9982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:22.555860 9982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:22.555918 9982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:22.555976 9982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:22.561326 9982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:22.562355 9982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:22.563416 9982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:22.573084 9988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.573169 9982 server_base.cc:1034] running on GCE node
W20260620 06:02:22.573076 9990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:22.573098 9987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.573794 9982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:22.574460 9982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:22.575657 9982 hybrid_clock.cc:648] HybridClock initialized: now 1781935342575641 us; error 29 us; skew 500 ppm
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:22.578645 9982 init.cc:377] Logged in from keytab as oryx/127.3.115.3@KRBTEST.COM (short username oryx)
I20260620 06:02:22.579558 9982 webserver.cc:460] Webserver started at http://127.3.115.3:46399/ using document root <none> and password file <none>
I20260620 06:02:22.580134 9982 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:22.580227 9982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:22.580471 9982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:22.582545 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "69a41d09bee54c1194739c2405386a2c"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "d3d3c368ea30225c9fb73f02f7fe7894"
server_key_iv: "dfa3188b5f05c433422535074a48eaab"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.583115 9982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "69a41d09bee54c1194739c2405386a2c"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "d3d3c368ea30225c9fb73f02f7fe7894"
server_key_iv: "dfa3188b5f05c433422535074a48eaab"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.586877 9982 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260620 06:02:22.589356 9997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.590620 9982 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:22.590766 9982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "69a41d09bee54c1194739c2405386a2c"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "d3d3c368ea30225c9fb73f02f7fe7894"
server_key_iv: "dfa3188b5f05c433422535074a48eaab"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.590880 9982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:22.622720 9982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:22.625916 9982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:22.626137 9982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:22.626746 9982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:22.627607 9982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:22.627681 9982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.627763 9982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:22.627815 9982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.637064 9982 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:35649
I20260620 06:02:22.637081 10110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:35649 every 8 connection(s)
I20260620 06:02:22.638070 9982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:22.644508 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 9982
I20260620 06:02:22.644677 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:22.645074 3532 external_mini_cluster.cc:1428] Setting key f9f9e942c01a0876b59d1528ddd452be
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:22.652510 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.640217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:36745 (local address 127.3.115.62:35659)
0620 06:02:22.640360 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.640363 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.641521 (+ 1158us) server_negotiation.cc:407] Connection header received
0620 06:02:22.642457 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.642460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.642513 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.642597 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.643807 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.644370 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.646001 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.646317 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.646437 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.649083 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.649087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.649089 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.649114 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.650411 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.650913 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.650915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.650917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.650998 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.651374 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.651377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.651378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.651600 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.651699 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.652203 (+ 504us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.652338 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:02:22.653389 10113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.640515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:36745)
0620 06:02:22.641327 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.641374 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.642208 (+ 834us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.642750 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.642757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.643140 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.643655 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.643671 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.644528 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.644538 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.645861 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.645873 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.646489 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.646494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.646688 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.647321 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.647337 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.648922 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.650518 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.650523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.650536 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.650809 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.651076 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.651078 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.651079 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.651235 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.651709 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.651714 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.651978 (+ 264us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.652136 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.652478 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":205,"threads_started":1}
I20260620 06:02:22.654588 10111 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35659
I20260620 06:02:22.654827 10111 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:22.655365 10111 heartbeater.cc:511] Master 127.3.115.62:35659 requested a full tablet report, sending...
I20260620 06:02:22.656404 9652 ts_manager.cc:194] Registered new tserver with Master: 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649)
I20260620 06:02:22.657055 9652 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:36745
I20260620 06:02:22.659169 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.61.keytab.
Entry for principal oryx/127.3.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.61.keytab.
I20260620 06:02:22.685899 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:35185
--webserver_interface=127.3.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.61.keytab
--principal=oryx/127.3.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:35659,127.3.115.61:35185
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:22.797854 10116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:22.798127 10116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:22.798206 10116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:22.805207 10116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:22.805302 10116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:22.805333 10116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:22.805367 10116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:22.805398 10116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:22.811791 10116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:35659,127.3.115.61:35185
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.61.keytab
--principal=oryx/127.3.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:35185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
W20260620 06:02:22.812006 10116 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.115.62:35659,127.3.115.61:35185'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 06:02:22.812945 10116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:22.814289 10116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:22.823186 10122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:22.823242 10124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.823236 10116 server_base.cc:1034] running on GCE node
W20260620 06:02:22.823436 10121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:22.824090 10116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:22.825299 10116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:22.826494 10116 hybrid_clock.cc:648] HybridClock initialized: now 1781935342826479 us; error 29 us; skew 500 ppm
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:22.829408 10116 init.cc:377] Logged in from keytab as oryx/127.3.115.61@KRBTEST.COM (short username oryx)
I20260620 06:02:22.830258 10116 webserver.cc:460] Webserver started at http://127.3.115.61:34165/ using document root <none> and password file <none>
I20260620 06:02:22.830772 10116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:22.830830 10116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:22.830991 10116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:22.833043 10116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/instance:
uuid: "051b1fc1c0a54686902499589658fb1d"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "aa05934291fceed7fb995a52114d3c3b"
server_key_iv: "c0a0f46490cfcd588453ae3023ebcc9d"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.833599 10116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal/instance:
uuid: "051b1fc1c0a54686902499589658fb1d"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "aa05934291fceed7fb995a52114d3c3b"
server_key_iv: "c0a0f46490cfcd588453ae3023ebcc9d"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.837486 10116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:02:22.839824 10131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.841089 10116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260620 06:02:22.841221 10116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
uuid: "051b1fc1c0a54686902499589658fb1d"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "aa05934291fceed7fb995a52114d3c3b"
server_key_iv: "c0a0f46490cfcd588453ae3023ebcc9d"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.841320 10116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:22.857723 10116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:22.860764 10116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:22.860951 10116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:22.867727 10116 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.61:35185
I20260620 06:02:22.867998 10183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.61:35185 every 8 connection(s)
I20260620 06:02:22.868854 10116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
I20260620 06:02:22.871635 10184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:22.874377 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 10116
I20260620 06:02:22.875561 10184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.61@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
I20260620 06:02:22.888731 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.877607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57002 (local address 127.3.115.62:35659)
0620 06:02:22.877722 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.877727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.878080 (+ 353us) server_negotiation.cc:407] Connection header received
0620 06:02:22.878272 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.878278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.878332 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.878387 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.880002 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.880856 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.882181 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.882416 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.882529 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.885076 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.885079 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.885081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.885105 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.886715 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.887221 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.887226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.887228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.887309 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.887645 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.887649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.887651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.887859 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.887970 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.888415 (+ 445us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.888532 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:02:22.889528 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.877560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57002)
0620 06:02:22.877960 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.878004 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.878161 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.878644 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.878656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.879117 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.879848 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.879860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.880991 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.880994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.882065 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.882074 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.882563 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.882567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.882688 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.883332 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.883356 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.884917 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.886839 (+ 1922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.886847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.886856 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.887097 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.887402 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.887407 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.887409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.887543 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.887966 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.887972 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.888203 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.888376 (+ 173us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.888707 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":112,"thread_start_us":99,"threads_started":1}
I20260620 06:02:22.889528 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.875928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60228 (local address 127.3.115.61:35185)
0620 06:02:22.876392 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.876410 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.876452 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:02:22.877296 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.877315 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.877876 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.879106 (+ 1230us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.879652 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.880703 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.881877 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.882443 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.882694 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.885477 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.885483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.885493 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.885512 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.886991 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.887429 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.887443 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.887444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.887541 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.887829 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.887834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.887835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.888196 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.888410 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.888615 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.888771 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":130,"server-negotiator.queue_time_us":320,"spinlock_wait_cycles":11136,"thread_start_us":148,"threads_started":1}
I20260620 06:02:22.891129 10184 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935342, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.61@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
I20260620 06:02:22.898901 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.891615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60242)
0620 06:02:22.891782 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.891798 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.891872 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.892062 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.892064 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.892215 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.892297 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.892301 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.893355 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.893357 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.893848 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.893855 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.894265 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.894268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.894457 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.894852 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.894862 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.896201 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.897701 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.897704 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.897705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.897885 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.898104 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.898106 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.898108 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.898174 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.898483 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.898485 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.898558 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.898644 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.898738 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":94}
I20260620 06:02:22.898916 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.891707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60242 (local address 127.3.115.61:35185)
0620 06:02:22.891812 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.891817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.891833 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:02:22.891939 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.891941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.891973 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.892036 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.892391 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.893263 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.893941 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.894177 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.894288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.896318 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.896321 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.896355 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.896381 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.897600 (+ 1219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.897963 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.897965 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.897967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.898030 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.898244 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.898246 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.898247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.898394 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.898466 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.898706 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.898789 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":38}
I20260620 06:02:22.901861 10184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Bootstrap starting.
I20260620 06:02:22.904269 10184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:22.905110 10184 log.cc:824] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:22.907156 10184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: No bootstrap required, opened a new log
I20260620 06:02:22.910339 10184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } }
I20260620 06:02:22.910583 10184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:22.910660 10184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 051b1fc1c0a54686902499589658fb1d, State: Initialized, Role: FOLLOWER
I20260620 06:02:22.911111 10184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } }
I20260620 06:02:22.911571 10190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } } }
I20260620 06:02:22.911680 10190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:22.912308 10184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:02:22.916827 10201 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:02:22.916913 10201 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:02:22.919111 10116 master_runner.cc:428] Detected that this master 051b1fc1c0a54686902499589658fb1d is joining an existing cluster
I20260620 06:02:22.919181 10116 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.115.61:35185
I20260620 06:02:22.929432 10208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.921757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60256)
0620 06:02:22.922030 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.922043 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.922180 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.922437 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.922439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.922704 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.922814 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.922819 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.923525 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.923531 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.924123 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.924130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.924566 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.924569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.924687 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.925222 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.925235 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.925549 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.927797 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.927803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.927807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.927996 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.928291 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.928294 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.928296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.928369 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.928841 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.928846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.928954 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.929086 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.929226 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260620 06:02:22.929759 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.921558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60256 (local address 127.3.115.61:35185)
0620 06:02:22.921680 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.921684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.922126 (+ 442us) server_negotiation.cc:407] Connection header received
0620 06:02:22.922287 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.922290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.922334 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.922414 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.922921 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.923421 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.924244 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.924451 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.924560 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.925697 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.925701 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.925703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.925724 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.927680 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.928112 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.928117 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.928119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.928194 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.928462 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.928465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.928467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.928714 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.928825 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.929151 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.929294 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":41}
I20260620 06:02:22.930325 10207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.921555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57010)
0620 06:02:22.921935 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.921950 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.922066 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.922441 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.922444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.922743 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.922867 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.922875 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.923958 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.923962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.924905 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.924915 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.925620 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.925623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.925716 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.926133 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.926147 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.926634 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.928613 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.928616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.928618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.928871 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.929187 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.929189 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.929191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.929282 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.929759 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.929762 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.929878 (+ 116us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.930027 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.930171 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":95,"threads_started":1}
I20260620 06:02:22.930370 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.921645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.3.115.62:35659)
0620 06:02:22.921764 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.921768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.922034 (+ 266us) server_negotiation.cc:407] Connection header received
0620 06:02:22.922189 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.922192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.922242 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.922294 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.923002 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.923817 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.925062 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.925394 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.925470 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.926789 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.926795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.926800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.926837 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.928510 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.928972 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.928977 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.928978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.929082 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.929389 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.929393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.929395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.929614 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.929680 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.930104 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.930222 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 06:02:22 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.62@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
I20260620 06:02:22.944579 10187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.935256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60262 (local address 127.3.115.61:35185)
0620 06:02:22.935373 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.935376 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.935387 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:02:22.935488 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.935492 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.935544 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.935608 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.936381 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.936900 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.938114 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.938351 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.938431 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.940840 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.940843 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.940845 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.940867 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.942378 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.942907 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.942910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.942912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.942998 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.943542 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.943545 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.943547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.943754 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.943853 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.944297 (+ 444us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.944410 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:02:22.945098 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.934550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60262)
0620 06:02:22.934905 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.934964 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.935092 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.935665 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.935672 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.936146 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.936271 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.936278 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.936997 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.937000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.937989 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.937999 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.938460 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.938464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.938575 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.939063 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.939079 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.940706 (+ 1627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.942481 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.942490 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.942497 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.942791 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.943331 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.943335 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.943337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.943438 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.943882 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.943892 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.944060 (+ 168us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.944228 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.944486 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":73,"threads_started":1}
I20260620 06:02:22.946879 9652 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.115.61:35185: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "051b1fc1c0a54686902499589658fb1d"
member_type: NON_VOTER
last_known_addr {
host: "127.3.115.61"
port: 35185
}
attrs {
promote: true
}
}
dest_uuid: "7804e9288f6f4e179d4d76ffb820f536"
cas_config_opid_index: -1
I20260620 06:02:22.947593 9652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } }
I20260620 06:02:22.949774 10212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 051b1fc1c0a54686902499589658fb1d (127.3.115.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } }
I20260620 06:02:22.950717 10158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:22.952178 10158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Refusing update from remote peer 7804e9288f6f4e179d4d76ffb820f536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 06:02:22.952347 10212 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.115.61:35185
I20260620 06:02:22.952934 10116 master.cc:537] Master@127.3.115.61:35185 shutting down...
I20260620 06:02:22.953410 10213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } }
I20260620 06:02:22.953506 10213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:22.953269 10215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:02:22.955506 9636 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 06:02:22.956453 9636 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 -> Peer 051b1fc1c0a54686902499589658fb1d (127.3.115.61:35185): Couldn't send request to peer 051b1fc1c0a54686902499589658fb1d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.3.115.60@KRBTEST.COM; defaulting to no policy
I20260620 06:02:22.957832 10116 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:22.958184 10116 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:22.958323 10116 tablet_replica.cc:330] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: stopping tablet replica
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.60@KRBTEST.COM" created.
I20260620 06:02:22.963866 10116 master.cc:552] Master@127.3.115.61:35185 shutdown complete.
I20260620 06:02:22.963976 10116 master_runner.cc:305] Clearing existing system tablet
I20260620 06:02:22.964308 10116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:22.964401 10116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:22.965808 10116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:22.966791 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:22.966988 10116 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:22.967252 10116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:22.967329 10116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
uuid: "051b1fc1c0a54686902499589658fb1d"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "aa05934291fceed7fb995a52114d3c3b"
server_key_iv: "c0a0f46490cfcd588453ae3023ebcc9d"
server_key_version: "encryptionkey@0"
I20260620 06:02:22.969575 10116 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:22.972352 10116 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:02:22.972441 10116 log.cc:1196] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 06:02:22.973265 10116 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Deleting consensus metadata
I20260620 06:02:22.973448 10116 master_runner.cc:315] Copying system tablet from 127.3.115.62:35659
I20260620 06:02:22.974611 10116 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: tablet copy: Beginning tablet copy session from remote peer at address 127.3.115.62:35659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.60.keytab.
Entry for principal oryx/127.3.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.60.keytab.
I20260620 06:02:22.976482 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:40179
--webserver_interface=127.3.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.60.keytab
--principal=oryx/127.3.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:02:22.983600 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.975237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57024 (local address 127.3.115.62:35659)
0620 06:02:22.975357 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:22.975361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:22.975526 (+ 165us) server_negotiation.cc:407] Connection header received
0620 06:02:22.975669 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:22.975672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:22.975723 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:22.975825 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:22.976387 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.977232 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.978346 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.978572 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.978681 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.980274 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:22.980277 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:22.980280 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:22.980314 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:22.981907 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.982362 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.982366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.982368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.982451 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:22.982736 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:22.982740 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:22.982742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:22.982929 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:22.983039 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:22.983319 (+ 280us) server_negotiation.cc:299] Negotiation successful
0620 06:02:22.983429 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:02:22.983613 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:22.975113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57024)
0620 06:02:22.975416 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:22.975433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:22.975565 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:22.975879 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:22.975882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:22.976098 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:22.976251 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.976262 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.977362 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.977368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.978208 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:22.978217 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:22.978687 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:22.978691 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:22.978792 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:22.979739 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:22.979752 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:22.980104 (+ 352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:22.982034 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.982037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.982040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.982255 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.982548 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:22.982551 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:22.982554 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:22.982639 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:22.983028 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:22.983031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:22.983138 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:02:22.983263 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:02:22.983422 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":138,"threads_started":1}
I20260620 06:02:22.984525 9672 tablet_copy_service.cc:140] P 7804e9288f6f4e179d4d76ffb820f536: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 051b1fc1c0a54686902499589658fb1d ({username='oryx', principal='oryx/127.3.115.61@KRBTEST.COM'} at 127.0.0.1:57024)
I20260620 06:02:22.984750 9672 tablet_copy_service.cc:161] P 7804e9288f6f4e179d4d76ffb820f536: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 051b1fc1c0a54686902499589658fb1d at {username='oryx', principal='oryx/127.3.115.61@KRBTEST.COM'} at 127.0.0.1:57024: session id = 051b1fc1c0a54686902499589658fb1d-00000000000000000000000000000000
I20260620 06:02:22.986172 9672 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:02:22.987715 10116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:02:22.989697 10116 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: tablet copy: Starting download of 0 data blocks...
I20260620 06:02:22.989914 10116 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: tablet copy: Starting download of 1 WAL segments...
I20260620 06:02:22.991556 10116 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:02:22.993194 9672 tablet_copy_service.cc:342] P 7804e9288f6f4e179d4d76ffb820f536: Request end of tablet copy session 051b1fc1c0a54686902499589658fb1d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.115.61@KRBTEST.COM'} at 127.0.0.1:57024
I20260620 06:02:22.993323 9672 tablet_copy_service.cc:433] P 7804e9288f6f4e179d4d76ffb820f536: ending tablet copy session 051b1fc1c0a54686902499589658fb1d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 051b1fc1c0a54686902499589658fb1d
I20260620 06:02:23.000226 10116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:23.001963 10228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.002053 10229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.002585 10116 server_base.cc:1034] running on GCE node
W20260620 06:02:23.004805 10231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.005033 10116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:23.005385 10116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:23.006526 10116 hybrid_clock.cc:648] HybridClock initialized: now 1781935343006527 us; error 51 us; skew 500 ppm
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:23.008034 10116 init.cc:377] Logged in from keytab as oryx/127.3.115.61@KRBTEST.COM (short username oryx)
I20260620 06:02:23.008531 10116 webserver.cc:460] Webserver started at http://127.3.115.61:41359/ using document root <none> and password file <none>
I20260620 06:02:23.008775 10116 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.008831 10116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.009933 10116 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:23.010881 10238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.011108 10116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:23.011179 10116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
uuid: "051b1fc1c0a54686902499589658fb1d"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "aa05934291fceed7fb995a52114d3c3b"
server_key_iv: "c0a0f46490cfcd588453ae3023ebcc9d"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.011262 10116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.031170 10116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:23.033259 10116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:23.038455 10116 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.61:35185
I20260620 06:02:23.038673 10290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.61:35185 every 8 connection(s)
I20260620 06:02:23.039072 10116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
I20260620 06:02:23.040563 10291 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:02:23.041491 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Bootstrap starting.
I20260620 06:02:23.049659 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:02:23.049976 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: Bootstrap complete.
I20260620 06:02:23.050513 10291 raft_consensus.cc:348] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } }
I20260620 06:02:23.050642 10291 raft_consensus.cc:729] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 051b1fc1c0a54686902499589658fb1d, State: Initialized, Role: LEARNER
I20260620 06:02:23.050752 10291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } }
I20260620 06:02:23.051165 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } }
I20260620 06:02:23.051257 10291 sys_catalog.cc:564] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:23.051342 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: LEARNER
I20260620 06:02:23.054059 10305 catalog_manager.cc:1259] Loaded cluster ID: ab969c50d19740719686c797238742e6
I20260620 06:02:23.054349 10305 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: loading cluster ID for follower catalog manager: success
I20260620 06:02:23.055943 10305 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: acquiring CA information for follower catalog manager: success
I20260620 06:02:23.057129 10305 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 06:02:23.127538 10227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:23.127825 10227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:23.127888 10227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:23.131932 10227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:23.132002 10227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:23.132022 10227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:23.132040 10227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:23.132055 10227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:23.137843 10227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.60.keytab
--principal=oryx/127.3.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:23.138875 10227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:23.139966 10227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:23.148692 10310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.148823 10311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.148697 10313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.149102 10227 server_base.cc:1034] running on GCE node
I20260620 06:02:23.149650 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:23.150802 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:23.152001 10227 hybrid_clock.cc:648] HybridClock initialized: now 1781935343151982 us; error 32 us; skew 500 ppm
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:23.154971 10227 init.cc:377] Logged in from keytab as oryx/127.3.115.60@KRBTEST.COM (short username oryx)
I20260620 06:02:23.155908 10227 webserver.cc:460] Webserver started at http://127.3.115.60:33627/ using document root <none> and password file <none>
I20260620 06:02:23.156559 10227 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.156652 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.156955 10227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:23.159106 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/instance:
uuid: "e9f073c9072d49fba384d895994c557c"
format_stamp: "Formatted at 2026-06-20 06:02:23 on dist-test-slave-4h14"
server_key: "34c48dcccd785db5902440c3fe59445b"
server_key_iv: "3cac2c2a8d91e077ba86972f04f8e6d2"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.159828 10227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal/instance:
uuid: "e9f073c9072d49fba384d895994c557c"
format_stamp: "Formatted at 2026-06-20 06:02:23 on dist-test-slave-4h14"
server_key: "34c48dcccd785db5902440c3fe59445b"
server_key_iv: "3cac2c2a8d91e077ba86972f04f8e6d2"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.163798 10227 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260620 06:02:23.166313 10320 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.167531 10227 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:23.167673 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
uuid: "e9f073c9072d49fba384d895994c557c"
format_stamp: "Formatted at 2026-06-20 06:02:23 on dist-test-slave-4h14"
server_key: "34c48dcccd785db5902440c3fe59445b"
server_key_iv: "3cac2c2a8d91e077ba86972f04f8e6d2"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.167788 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.181847 9843 heartbeater.cc:503] Master 127.3.115.62:35659 was elected leader, sending a full tablet report...
I20260620 06:02:23.190435 10227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:23.193511 10227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:23.193722 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:23.200752 10227 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.60:40179
I20260620 06:02:23.200803 10372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.60:40179 every 8 connection(s)
I20260620 06:02:23.201750 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
I20260620 06:02:23.203617 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 10227
I20260620 06:02:23.205629 10373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:23.210090 10373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.115.60@KRBTEST.COM
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.60@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:23.218570 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.205128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.3.115.60:40179)
0620 06:02:23.205512 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.205527 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.205562 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:02:23.206275 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.206294 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.206650 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.206936 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.207851 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.208933 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.210058 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.210612 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.210742 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.213815 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.213821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.213831 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.213857 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.215581 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.216129 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.216140 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.216145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.216266 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.216567 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.216571 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.216573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.217025 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.217253 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.217555 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.217824 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"spinlock_wait_cycles":768,"thread_start_us":122,"threads_started":1}
I20260620 06:02:23.221146 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.212222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.3.115.62:35659)
0620 06:02:23.212357 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.212362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.212626 (+ 264us) server_negotiation.cc:407] Connection header received
0620 06:02:23.212883 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.212886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.212930 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.213024 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.213760 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.214275 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.215161 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.215388 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.215501 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.217611 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.217614 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.217617 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.217651 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.219251 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.219798 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.219801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.219803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.219896 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.220220 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.220222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.220223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.220393 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.220473 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.220867 (+ 394us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.220983 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:02:23.221181 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.211987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57026)
0620 06:02:23.212498 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.212556 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.212721 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.213072 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.213077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.213501 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.213630 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.213636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.214407 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.214411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.215037 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.215045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.215501 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.215504 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.215626 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.216100 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.216111 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.217469 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.219361 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.219371 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.219377 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.219672 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.219998 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.220001 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.220003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.220097 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.220494 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.220502 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.220626 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.220810 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.220984 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":198,"threads_started":1}
I20260620 06:02:23.222350 10373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.60@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
I20260620 06:02:23.229918 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.222830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60278)
0620 06:02:23.223124 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.223136 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.223195 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.223403 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.223405 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.223529 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.223614 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.223618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.224442 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.224444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.225044 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.225052 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.225462 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.225464 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.225535 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.225993 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.226008 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.227144 (+ 1136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.228616 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.228619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.228620 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.228891 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.229139 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.229141 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.229142 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.229212 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.229537 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.229539 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.229615 (+ 76us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.229692 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.229795 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231}
I20260620 06:02:23.229976 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.222926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.3.115.61:35185)
0620 06:02:23.223170 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.223174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.223187 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:02:23.223270 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.223273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.223311 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.223378 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.223708 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.224356 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.225150 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.225368 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.225451 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.227270 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.227273 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.227275 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.227297 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.228523 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.228978 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.228980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.228981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.229059 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.229292 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.229294 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.229295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.229460 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.229536 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.229746 (+ 210us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.229846 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260620 06:02:23.230907 10373 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.60@KRBTEST.COM for oryx/127.3.115.60@KRBTEST.COM
I20260620 06:02:23.239053 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.231921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.0.0.1:55516)
0620 06:02:23.232017 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.232030 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.232101 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.232305 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.232307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.232434 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.232504 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.232509 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.233301 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.233304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.233794 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.233801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.234308 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.234311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.234415 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.234855 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.234865 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.236134 (+ 1269us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.237855 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.237858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.237859 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.238033 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.238268 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.238270 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.238271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.238339 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.238649 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.238651 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.238724 (+ 73us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.238804 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.238909 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 06:02:23.239084 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.231492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.3.115.60:40179)
0620 06:02:23.231646 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.231651 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.232109 (+ 458us) server_negotiation.cc:407] Connection header received
0620 06:02:23.232178 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.232180 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.232215 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.232281 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.232596 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.233190 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.233911 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.234201 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.234295 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.236255 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.236257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.236260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.236280 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.237760 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.238113 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.238116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.238117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.238204 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.238413 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.238415 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.238416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.238573 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.238649 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.238846 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.238963 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":58}
I20260620 06:02:23.242244 10373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Bootstrap starting.
I20260620 06:02:23.244621 10373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:23.245428 10373 log.cc:824] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:23.247447 10373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: No bootstrap required, opened a new log
I20260620 06:02:23.250563 10373 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } }
I20260620 06:02:23.250809 10373 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:23.250891 10373 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9f073c9072d49fba384d895994c557c, State: Initialized, Role: FOLLOWER
I20260620 06:02:23.251487 10373 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } }
I20260620 06:02:23.252004 10379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } } }
I20260620 06:02:23.252204 10379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:23.252955 10373 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:02:23.257900 10390 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:02:23.257992 10390 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:02:23.261173 10227 master_runner.cc:428] Detected that this master e9f073c9072d49fba384d895994c557c is joining an existing cluster
I20260620 06:02:23.261248 10227 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.115.60:40179
I20260620 06:02:23.272118 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9714
I20260620 06:02:23.273186 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.263748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.3.115.62:35659)
0620 06:02:23.263860 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.263863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.263875 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:02:23.263931 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.263935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.263987 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.264080 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.264783 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.265591 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.266833 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.267166 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.267250 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.268769 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.268771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.268773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.268816 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.270474 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.271838 (+ 1364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.271842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.271844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.271929 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.272264 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.272267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.272269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.272454 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.272513 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.272884 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.273026 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 06:02:23.273423 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.263661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.3.115.61:35185)
0620 06:02:23.263782 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.263787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.264413 (+ 626us) server_negotiation.cc:407] Connection header received
0620 06:02:23.264582 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.264587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.264637 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.264711 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.265407 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.266181 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.267681 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.267998 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.268077 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.269534 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.269546 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.269549 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.269576 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.271270 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.271804 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.271808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.271810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.271923 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.272269 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.272275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.272280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.272534 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.272608 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.273026 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.273173 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:02:23.273914 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.263272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57030)
0620 06:02:23.263585 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.263603 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.263699 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.264213 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.264216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.264460 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.264628 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.264635 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.265717 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.265720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.266695 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.266707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.267410 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.267413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.267507 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.268052 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.268067 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.268535 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.271376 (+ 2841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.271380 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.271382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.271721 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.272034 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.272037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.272039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.272119 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.272585 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.272588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.272710 (+ 122us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.273411 (+ 701us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.273734 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":27,"thread_start_us":155,"threads_started":1}
I20260620 06:02:23.274148 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.263334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60282)
0620 06:02:23.264328 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.264342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.264471 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.264859 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.264862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.265131 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.265278 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.265285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.266311 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.266314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.267553 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.267561 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.268219 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.268224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.268407 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.268927 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.268942 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.269393 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.271359 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.271362 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.271366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.271694 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.272058 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.272061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.272062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.272152 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.272723 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.272729 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.272833 (+ 104us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.273561 (+ 728us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.273836 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"mutex_wait_us":187,"thread_start_us":87,"threads_started":1}
I20260620 06:02:23.280319 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:45025
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:35659
--webserver_port=41865
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:02:23.289188 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.264004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.0.0.1:55522)
0620 06:02:23.264398 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.264412 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.264551 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.264878 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.264880 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.265173 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.265320 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.265328 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.266300 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.266303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.267231 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.267242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.268221 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.268224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.268397 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.285140 (+ 16743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.285160 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.285669 (+ 509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.287656 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.287662 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.287667 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.287921 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.288185 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.288188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.288190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.288268 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.288679 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.288682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.288770 (+ 88us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.288861 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.288983 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":84,"spinlock_wait_cycles":9600,"thread_start_us":86,"threads_started":1}
I20260620 06:02:23.289710 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.264064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.3.115.60:40179)
0620 06:02:23.264160 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.264167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.264475 (+ 308us) server_negotiation.cc:407] Connection header received
0620 06:02:23.264649 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.264669 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.264707 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.264758 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.265439 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.266181 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.267350 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.267674 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.268088 (+ 414us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.285838 (+ 17750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.285846 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.285848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.285874 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.287540 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.288020 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.288023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.288026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.288101 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.288359 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.288364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.288366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.288538 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.288623 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.289476 (+ 853us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.289579 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935341, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.62@KRBTEST.COM for oryx/127.3.115.60@KRBTEST.COM
I20260620 06:02:23.296087 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.284695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.0.0.1:55524)
0620 06:02:23.285697 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.285713 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.285795 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.287703 (+ 1908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.287709 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.287906 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.288044 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.288053 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.289226 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.289232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.290071 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.290081 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.290670 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.290673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.290775 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.291260 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.291274 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.292462 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.294424 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.294428 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.294430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.294631 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.294972 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.294976 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.294978 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.295091 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.295544 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.295547 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.295649 (+ 102us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.295773 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.295936 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":915}
I20260620 06:02:23.296103 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.284959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55524 (local address 127.3.115.60:40179)
0620 06:02:23.287401 (+ 2442us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.287412 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.287428 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:02:23.287505 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.287509 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.287559 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.288154 (+ 595us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.288277 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.289105 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.290213 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.290536 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.290663 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.292616 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.292620 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.292623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.292753 (+ 130us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.294308 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.294739 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.294742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.294744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.294879 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.295182 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.295187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.295189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.295441 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.295554 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.295825 (+ 271us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.295934 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2360,"thread_start_us":91,"threads_started":1}
I20260620 06:02:23.297168 9652 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.115.60:40179: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e9f073c9072d49fba384d895994c557c"
member_type: NON_VOTER
last_known_addr {
host: "127.3.115.60"
port: 40179
}
attrs {
promote: true
}
}
dest_uuid: "7804e9288f6f4e179d4d76ffb820f536"
cas_config_opid_index: 5
I20260620 06:02:23.297884 9652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } }
I20260620 06:02:23.299149 10214 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e9f073c9072d49fba384d895994c557c (127.3.115.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.300678 10214 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.115.60:40179
I20260620 06:02:23.300884 10214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.300983 10214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.301697 10345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:23.303097 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.299281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60288 (local address 127.3.115.61:35185)
0620 06:02:23.299408 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.299412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.299428 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:02:23.299484 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.299490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.299562 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.299673 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:23.299934 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.300692 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.302086 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.302567 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.302698 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.302884 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.302922 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.302990 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:02:23.303097 10209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.299166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60288)
0620 06:02:23.299304 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.299318 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.299387 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.299730 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.299733 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.299748 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:23.299839 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.299844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.300870 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.300873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.301956 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.301965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.302696 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.302700 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.302783 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.302830 (+ 47us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.302897 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.302973 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260620 06:02:23.303640 10345 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Refusing update from remote peer 7804e9288f6f4e179d4d76ffb820f536: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:02:23.304072 10227 master.cc:537] Master@127.3.115.60:40179 shutting down...
I20260620 06:02:23.304309 10215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:23.305418 10265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 LEARNER]: Refusing update from remote peer 7804e9288f6f4e179d4d76ffb820f536: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 06:02:23.305871 10215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260620 06:02:23.306517 9638 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 -> Peer e9f073c9072d49fba384d895994c557c (127.3.115.60:40179): Couldn't send request to peer e9f073c9072d49fba384d895994c557c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:02:23.307804 10265 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e9f073c9072d49fba384d895994c557c (127.3.115.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.310262 10214 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: attempting to promote NON_VOTER 051b1fc1c0a54686902499589658fb1d to VOTER
I20260620 06:02:23.310410 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7804e9288f6f4e179d4d76ffb820f536. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.310518 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: LEARNER
I20260620 06:02:23.310611 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: NON_VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: true } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.310674 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: LEARNER
I20260620 06:02:23.310788 10214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } }
I20260620 06:02:23.312036 10265 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 LEARNER]: Refusing update from remote peer 7804e9288f6f4e179d4d76ffb820f536: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 06:02:23.312451 10215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260620 06:02:23.312577 10227 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 06:02:23.312634 9638 consensus_peers.cc:487] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 -> Peer e9f073c9072d49fba384d895994c557c (127.3.115.60:40179): Couldn't send request to peer e9f073c9072d49fba384d895994c557c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:02:23.312930 10227 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:23.313035 10227 tablet_replica.cc:330] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: stopping tablet replica
I20260620 06:02:23.314011 10213 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 051b1fc1c0a54686902499589658fb1d (127.3.115.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.315547 10215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.315580 10213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.315670 10215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.315696 10213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.316126 10264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 051b1fc1c0a54686902499589658fb1d (127.3.115.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.317646 10294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.317771 10294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:23.318819 10227 master.cc:552] Master@127.3.115.60:40179 shutdown complete.
I20260620 06:02:23.318863 10227 master_runner.cc:305] Clearing existing system tablet
I20260620 06:02:23.319098 10227 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.319142 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.320179 10227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:23.321012 10404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.321209 10227 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.321343 10227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:23.321381 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
uuid: "e9f073c9072d49fba384d895994c557c"
format_stamp: "Formatted at 2026-06-20 06:02:23 on dist-test-slave-4h14"
server_key: "34c48dcccd785db5902440c3fe59445b"
server_key_iv: "3cac2c2a8d91e077ba86972f04f8e6d2"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.323082 10227 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:23.325227 10227 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 06:02:23.325325 10227 log.cc:1196] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 06:02:23.325577 10227 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Deleting consensus metadata
I20260620 06:02:23.325760 10227 master_runner.cc:315] Copying system tablet from 127.3.115.62:35659
I20260620 06:02:23.326928 10227 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: tablet copy: Beginning tablet copy session from remote peer at address 127.3.115.62:35659
I20260620 06:02:23.336051 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.327590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.3.115.62:35659)
0620 06:02:23.327724 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.327729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.327889 (+ 160us) server_negotiation.cc:407] Connection header received
0620 06:02:23.328018 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.328021 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.328074 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.328174 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.328734 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.329509 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.330704 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.331052 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.331150 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.332461 (+ 1311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.332465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.332468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.332495 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.334258 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.334739 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.334742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.334744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.334833 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.335115 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.335118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.335120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.335337 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.335435 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.335750 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.335881 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:02:23.336539 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.327484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57036)
0620 06:02:23.327789 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.327805 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.327919 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.328207 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.328210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.328431 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.328552 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.328559 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.329632 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.329636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.330566 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.330576 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.331187 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.331191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.331288 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.331867 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.331882 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.332306 (+ 424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.334372 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.334376 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.334378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.334634 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.334927 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.334930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.334933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.335024 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.335435 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.335438 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.335538 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.336204 (+ 666us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.336355 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260620 06:02:23.337093 9672 tablet_copy_service.cc:140] P 7804e9288f6f4e179d4d76ffb820f536: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e9f073c9072d49fba384d895994c557c ({username='oryx', principal='oryx/127.3.115.60@KRBTEST.COM'} at 127.0.0.1:57036)
I20260620 06:02:23.337198 9672 tablet_copy_service.cc:161] P 7804e9288f6f4e179d4d76ffb820f536: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e9f073c9072d49fba384d895994c557c at {username='oryx', principal='oryx/127.3.115.60@KRBTEST.COM'} at 127.0.0.1:57036: session id = e9f073c9072d49fba384d895994c557c-00000000000000000000000000000000
I20260620 06:02:23.338276 9672 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:02:23.339649 10227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 06:02:23.342145 10227 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: tablet copy: Starting download of 0 data blocks...
I20260620 06:02:23.342355 10227 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: tablet copy: Starting download of 1 WAL segments...
I20260620 06:02:23.343925 10227 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:02:23.345866 9672 tablet_copy_service.cc:342] P 7804e9288f6f4e179d4d76ffb820f536: Request end of tablet copy session e9f073c9072d49fba384d895994c557c-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.115.60@KRBTEST.COM'} at 127.0.0.1:57036
I20260620 06:02:23.345961 9672 tablet_copy_service.cc:433] P 7804e9288f6f4e179d4d76ffb820f536: ending tablet copy session e9f073c9072d49fba384d895994c557c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e9f073c9072d49fba384d895994c557c
I20260620 06:02:23.354138 10227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:23.355834 10411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.357506 10412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.357875 10227 server_base.cc:1034] running on GCE node
W20260620 06:02:23.357909 10414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.358271 10227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:23.358583 10227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:23.359797 10227 hybrid_clock.cc:648] HybridClock initialized: now 1781935343359784 us; error 31 us; skew 500 ppm
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:23.361590 10227 init.cc:377] Logged in from keytab as oryx/127.3.115.60@KRBTEST.COM (short username oryx)
I20260620 06:02:23.362061 10227 webserver.cc:460] Webserver started at http://127.3.115.60:35633/ using document root <none> and password file <none>
I20260620 06:02:23.362253 10227 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.362310 10227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.363698 10227 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:23.364789 10421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.365064 10227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:23.365139 10227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
uuid: "e9f073c9072d49fba384d895994c557c"
format_stamp: "Formatted at 2026-06-20 06:02:23 on dist-test-slave-4h14"
server_key: "34c48dcccd785db5902440c3fe59445b"
server_key_iv: "3cac2c2a8d91e077ba86972f04f8e6d2"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.365224 10227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.407501 10227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:23.409821 10227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:23.415038 10227 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.60:40179
I20260620 06:02:23.415069 10473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.60:40179 every 8 connection(s)
I20260620 06:02:23.415681 10227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
I20260620 06:02:23.417591 10474 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:02:23.418741 10474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Bootstrap starting.
I20260620 06:02:23.427716 9977 heartbeater.cc:503] Master 127.3.115.62:35659 was elected leader, sending a full tablet report...
I20260620 06:02:23.430606 10474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:02:23.430968 10474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Bootstrap complete.
I20260620 06:02:23.431556 10474 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } }
I20260620 06:02:23.431712 10474 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9f073c9072d49fba384d895994c557c, State: Initialized, Role: LEARNER
I20260620 06:02:23.431825 10474 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } }
I20260620 06:02:23.432200 10477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.432302 10474 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:23.432416 10477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: This master's current role is: LEARNER
I20260620 06:02:23.435837 10488 catalog_manager.cc:1259] Loaded cluster ID: ab969c50d19740719686c797238742e6
I20260620 06:02:23.435966 10488 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: loading cluster ID for follower catalog manager: success
I20260620 06:02:23.438244 10488 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: acquiring CA information for follower catalog manager: success
I20260620 06:02:23.439908 10488 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260620 06:02:23.442627 10399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:23.442952 10399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:23.443063 10399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:23.447459 10399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:23.447546 10399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:23.447667 10399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:23.453713 10399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:45025
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=41865
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:23.454861 10399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:23.455961 10399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:23.467036 10493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.466972 10494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.466971 10496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.468031 10399 server_base.cc:1034] running on GCE node
I20260620 06:02:23.468477 10399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:23.469260 10399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:23.470453 10399 hybrid_clock.cc:648] HybridClock initialized: now 1781935343470419 us; error 52 us; skew 500 ppm
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:23.473243 10399 init.cc:377] Logged in from keytab as oryx/127.3.115.1@KRBTEST.COM (short username oryx)
I20260620 06:02:23.474150 10399 webserver.cc:460] Webserver started at http://127.3.115.1:41865/ using document root <none> and password file <none>
I20260620 06:02:23.474731 10399 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.474848 10399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.478641 10399 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 06:02:23.480768 10503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.482012 10399 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:23.482148 10399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "22662410bfb74fa091248719a263f65f"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "fdbf09c6afee970d70bf7417d72e3897"
server_key_iv: "75cd20f71947a14db129d1b8374323c3"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.482695 10399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.506520 10399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:23.509999 10399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:23.510211 10399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:23.510838 10399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:23.511754 10399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:23.511826 10399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.511897 10399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:23.511948 10399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.521008 10399 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:45025
I20260620 06:02:23.521025 10616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:45025 every 8 connection(s)
I20260620 06:02:23.522044 10399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:23.525569 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 10399
I20260620 06:02:23.525777 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9848
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:23.533963 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:46583
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:35659
--webserver_port=33617
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.60@KRBTEST.COM
I20260620 06:02:23.539791 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.523911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:57125 (local address 127.3.115.62:35659)
0620 06:02:23.524053 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.524057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.524948 (+ 891us) server_negotiation.cc:407] Connection header received
0620 06:02:23.527742 (+ 2794us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.527745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.527801 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.527854 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.528436 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.529223 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530327 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.530641 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530715 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.535118 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.535122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.535124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.535156 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.537064 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.537855 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.537860 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.537863 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.537951 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538404 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.538410 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.538413 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.538645 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.538720 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.539498 (+ 778us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.539631 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:02:23.539906 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.523879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:58503 (local address 127.3.115.61:35185)
0620 06:02:23.524069 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.524074 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.524945 (+ 871us) server_negotiation.cc:407] Connection header received
0620 06:02:23.526204 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.526210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.526283 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.526340 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.527684 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.528466 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530208 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.530530 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530601 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.534854 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.534861 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.534867 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.534901 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.537139 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.537859 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.537865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.537870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.537962 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538413 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.538418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.538421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.538694 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.538791 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.539524 (+ 733us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.539671 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 06:02:23.540570 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.525502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37425 (local address 127.3.115.60:40179)
0620 06:02:23.528591 (+ 3089us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.528609 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.528627 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:02:23.528700 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.528704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.528761 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.528852 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.529535 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.530795 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.532060 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.532500 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.532616 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.535127 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.535131 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.535134 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.535644 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.538121 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538587 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.538591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.538592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.538676 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538973 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.538976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.538977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.539186 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.539278 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.540301 (+ 1023us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.540428 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2986,"thread_start_us":80,"threads_started":1}
I20260620 06:02:23.541049 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.524312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:57125)
0620 06:02:23.524795 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.524847 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.525926 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.527986 (+ 2060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.527989 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.528182 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.528308 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.528314 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.529373 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.529376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.530207 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.530214 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.530849 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530852 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.531140 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.532014 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.532025 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.534957 (+ 2932us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.537206 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.537213 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.537235 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.537729 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.538073 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538077 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.538079 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.538289 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.538800 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.538807 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.539204 (+ 397us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.540013 (+ 809us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.540133 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":131,"spinlock_wait_cycles":53760,"thread_start_us":136,"threads_started":1}
I20260620 06:02:23.541349 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.524312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.3.115.1:58503)
0620 06:02:23.524795 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.524846 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.526098 (+ 1252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.526717 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.526724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.527065 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.527539 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.527552 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.528620 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.528623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.530070 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.530081 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.530815 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530819 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.531142 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.531985 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.532007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.534653 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.537282 (+ 2629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.537288 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.537293 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.537739 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.538079 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538085 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.538089 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.538289 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.538857 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.538862 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.539218 (+ 356us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.540106 (+ 888us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.540265 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":101,"spinlock_wait_cycles":7168,"thread_start_us":127,"threads_started":1}
I20260620 06:02:23.541520 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.525389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.3.115.1:37425)
0620 06:02:23.525737 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.525753 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.527182 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.528994 (+ 1812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.528999 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.529219 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.529368 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.529379 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.530937 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.530943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.531920 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.531933 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.532798 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.532805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.532930 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.533467 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.533487 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.534957 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.538230 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.538235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.538483 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.538791 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.538794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.538796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.538876 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.539321 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.539323 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.539426 (+ 103us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.539538 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.539901 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"spinlock_wait_cycles":6400,"thread_start_us":91,"threads_started":1}
I20260620 06:02:23.542575 10618 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35659
I20260620 06:02:23.542579 10617 heartbeater.cc:347] Connected to a master server at 127.3.115.60:40179
I20260620 06:02:23.542583 10619 heartbeater.cc:347] Connected to a master server at 127.3.115.61:35185
I20260620 06:02:23.542850 10619 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.542848 10618 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.542848 10617 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.543471 10617 heartbeater.cc:511] Master 127.3.115.60:40179 requested a full tablet report, sending...
I20260620 06:02:23.543555 10618 heartbeater.cc:511] Master 127.3.115.62:35659 requested a full tablet report, sending...
I20260620 06:02:23.543474 10619 heartbeater.cc:511] Master 127.3.115.61:35185 requested a full tablet report, sending...
I20260620 06:02:23.545433 9652 ts_manager.cc:194] Re-registered known tserver with Master: 22662410bfb74fa091248719a263f65f (127.3.115.1:45025)
I20260620 06:02:23.545433 10437 ts_manager.cc:194] Registered new tserver with Master: 22662410bfb74fa091248719a263f65f (127.3.115.1:45025)
I20260620 06:02:23.545499 10255 ts_manager.cc:194] Registered new tserver with Master: 22662410bfb74fa091248719a263f65f (127.3.115.1:45025)
I20260620 06:02:23.546176 9652 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:57125
W20260620 06:02:23.653617 10625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:23.653903 10625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:23.653981 10625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:23.658100 10625 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:23.658172 10625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:23.658233 10625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:23.659389 10111 heartbeater.cc:503] Master 127.3.115.62:35659 was elected leader, sending a full tablet report...
I20260620 06:02:23.664206 10625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:46583
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=33617
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10625
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:23.665401 10625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:23.666491 10625 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:23.676044 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.676085 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.676031 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.676442 10625 server_base.cc:1034] running on GCE node
I20260620 06:02:23.676908 10625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:23.677546 10625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:23.678702 10625 hybrid_clock.cc:648] HybridClock initialized: now 1781935343678688 us; error 27 us; skew 500 ppm
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:23.681401 10625 init.cc:377] Logged in from keytab as oryx/127.3.115.2@KRBTEST.COM (short username oryx)
I20260620 06:02:23.682230 10625 webserver.cc:460] Webserver started at http://127.3.115.2:33617/ using document root <none> and password file <none>
I20260620 06:02:23.682730 10625 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.682808 10625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.686717 10625 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260620 06:02:23.688769 10640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.689994 10625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:23.690114 10625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "d09430c33e5d42eebc0403da38b85de9"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "79163e7bed62ff95c164ad2e9e545906"
server_key_iv: "afb32b74cdc646a791b7983046ab5ce9"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.690675 10625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.709165 10625 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:23.712328 10625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:23.712486 10625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:23.713068 10625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:23.713873 10625 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:23.713918 10625 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.713958 10625 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:23.713980 10625 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.723110 10625 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:46583
I20260620 06:02:23.723133 10753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:46583 every 8 connection(s)
I20260620 06:02:23.724119 10625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:23.731518 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 10625
I20260620 06:02:23.731691 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9982
I20260620 06:02:23.737330 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:35649
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:35659
--webserver_port=46399
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.60@KRBTEST.COM
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
I20260620 06:02:23.745544 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.725906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:51039 (local address 127.3.115.62:35659)
0620 06:02:23.726060 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.726066 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.727653 (+ 1587us) server_negotiation.cc:407] Connection header received
0620 06:02:23.728227 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.728231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.728285 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.728338 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.729749 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.730529 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.732525 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.732911 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.732993 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.740574 (+ 7581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.740579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.740585 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.740627 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.742826 (+ 2199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743516 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.743519 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.743523 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.743614 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.744083 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.744089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.744091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.744347 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.744419 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.745237 (+ 818us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.745371 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:02:23.745544 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.725876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:43705 (local address 127.3.115.61:35185)
0620 06:02:23.726003 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.726007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.727470 (+ 1463us) server_negotiation.cc:407] Connection header received
0620 06:02:23.728197 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.728200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.728257 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.728364 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.729704 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.730520 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.737227 (+ 6707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.737614 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.737718 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.741482 (+ 3764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.741488 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.741493 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.741530 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.743443 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743903 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.743906 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.743908 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.743995 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.744320 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.744324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.744326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.744554 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.744627 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.745234 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.745371 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:02:23.746824 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.726408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.3.115.2:43705)
0620 06:02:23.727366 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.727378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.727998 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.728539 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.728546 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.729034 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.729561 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.729575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.731091 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.731094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.737068 (+ 5974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.737083 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.737924 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.737930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.738136 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.738903 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.738915 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.741303 (+ 2388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.743556 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.743566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.743795 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.744125 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.744130 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.744134 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.744206 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.744718 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.744720 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.744938 (+ 218us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.745758 (+ 820us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.745960 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":235,"spinlock_wait_cycles":3200,"thread_start_us":164,"threads_started":1}
I20260620 06:02:23.746833 10760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.726409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:51039)
0620 06:02:23.727562 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.727571 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.728003 (+ 432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.728540 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.728547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.729076 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.729571 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.729578 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.730636 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.730649 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.732377 (+ 1728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.732391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.733161 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.733163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.737258 (+ 4095us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.738319 (+ 1061us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.738343 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.740330 (+ 1987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.742976 (+ 2646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.742984 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.743006 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.743384 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.743734 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.743753 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.743948 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.744498 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.744506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.744950 (+ 444us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.745699 (+ 749us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.745872 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"mutex_wait_us":480,"spinlock_wait_cycles":13568,"thread_start_us":126,"threads_started":1}
I20260620 06:02:23.747773 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.726345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:41709 (local address 127.3.115.60:40179)
0620 06:02:23.726462 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.726468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.727047 (+ 579us) server_negotiation.cc:407] Connection header received
0620 06:02:23.728228 (+ 1181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.728231 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.728286 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.728338 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.729730 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.730271 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.737199 (+ 6928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.737637 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.737778 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.740878 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.740885 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.740891 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.740930 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.743136 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743719 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.743725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.743728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.743826 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.744201 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.744207 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.744212 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.744472 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.744576 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.747471 (+ 2895us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.747616 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:02:23.748219 10756 heartbeater.cc:347] Connected to a master server at 127.3.115.61:35185
I20260620 06:02:23.748229 10755 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35659
I20260620 06:02:23.748380 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.726408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.3.115.2:41709)
0620 06:02:23.726899 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.726949 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.728001 (+ 1052us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.728540 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.728547 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.729074 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.729570 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.729578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.730415 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.730418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.737019 (+ 6601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.737033 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.737960 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.737964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.738319 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.738743 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.738758 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.740708 (+ 1950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.743285 (+ 2577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.743296 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.743573 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.743958 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.743963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.743967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.744081 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.744627 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.744632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.744970 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.745164 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.745599 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":384,"spinlock_wait_cycles":12928,"thread_start_us":121,"threads_started":1}
I20260620 06:02:23.748497 10755 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.748497 10756 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.749174 10756 heartbeater.cc:511] Master 127.3.115.61:35185 requested a full tablet report, sending...
I20260620 06:02:23.749172 10755 heartbeater.cc:511] Master 127.3.115.62:35659 requested a full tablet report, sending...
I20260620 06:02:23.749775 10754 heartbeater.cc:347] Connected to a master server at 127.3.115.60:40179
I20260620 06:02:23.749873 10754 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.750093 10754 heartbeater.cc:511] Master 127.3.115.60:40179 requested a full tablet report, sending...
I20260620 06:02:23.750906 10437 ts_manager.cc:194] Registered new tserver with Master: d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583)
I20260620 06:02:23.751026 10255 ts_manager.cc:194] Registered new tserver with Master: d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583)
I20260620 06:02:23.752104 9652 ts_manager.cc:194] Re-registered known tserver with Master: d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583)
I20260620 06:02:23.752998 9652 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:51039
I20260620 06:02:23.833650 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.829623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.0.0.1:55528)
0620 06:02:23.829904 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.829916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.830020 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.830317 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.830319 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.830334 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:23.830431 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.830435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.831433 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.831435 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.832266 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.832276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.833238 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.833250 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.833322 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.833345 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.833398 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.833456 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":64,"threads_started":1}
I20260620 06:02:23.833638 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.829723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55528 (local address 127.3.115.60:40179)
0620 06:02:23.829858 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.829863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.829998 (+ 135us) server_negotiation.cc:407] Connection header received
0620 06:02:23.830118 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.830121 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.830187 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.830308 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:23.830532 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.831325 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.832398 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.833113 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.833230 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.833379 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.833417 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.833487 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:02:23.834288 10448 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 06:02:23.834679 10477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7804e9288f6f4e179d4d76ffb820f536. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.834851 10477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: This master's current role is: LEARNER
W20260620 06:02:23.858093 10761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:23.858352 10761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:23.858395 10761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:23.862484 10761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:23.862577 10761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:23.862687 10761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:23.868142 10761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:35649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=46399
--tserver_master_addrs=127.3.115.62:35659,127.3.115.61:35185,127.3.115.60:40179
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:23.869253 10761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:23.870244 10761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:23.879844 10769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.879844 10768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:23.879913 10771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:23.880157 10761 server_base.cc:1034] running on GCE node
I20260620 06:02:23.880568 10761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:23.881253 10761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:23.882416 10761 hybrid_clock.cc:648] HybridClock initialized: now 1781935343882402 us; error 27 us; skew 500 ppm
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:23.885250 10761 init.cc:377] Logged in from keytab as oryx/127.3.115.3@KRBTEST.COM (short username oryx)
I20260620 06:02:23.886229 10761 webserver.cc:460] Webserver started at http://127.3.115.3:46399/ using document root <none> and password file <none>
I20260620 06:02:23.886792 10761 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:23.886904 10761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:23.890784 10761 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260620 06:02:23.892975 10778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.894392 10761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:23.894523 10761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "69a41d09bee54c1194739c2405386a2c"
format_stamp: "Formatted at 2026-06-20 06:02:22 on dist-test-slave-4h14"
server_key: "d3d3c368ea30225c9fb73f02f7fe7894"
server_key_iv: "dfa3188b5f05c433422535074a48eaab"
server_key_version: "encryptionkey@0"
I20260620 06:02:23.895169 10761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:23.929977 10761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:23.933327 10761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:23.933575 10761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:23.934237 10761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:23.935200 10761 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:23.935253 10761 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.935317 10761 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:23.935348 10761 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:23.944046 10761 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:35649
I20260620 06:02:23.944067 10891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:35649 every 8 connection(s)
I20260620 06:02:23.945205 10761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:23.945413 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 10761
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.60@KRBTEST.COM
I20260620 06:02:23.960989 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.949062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60304 (local address 127.3.115.61:35185)
0620 06:02:23.949723 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.949736 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.950066 (+ 330us) server_negotiation.cc:407] Connection header received
0620 06:02:23.950111 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.950114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.950163 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.950249 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.951263 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.952127 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.953233 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.953594 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.953690 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.956275 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.956278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.956281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.956306 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.958793 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.959321 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.959325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.959327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.959411 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.959704 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.959707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.959709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.959975 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.960103 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.960677 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.960809 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"spinlock_wait_cycles":1280,"thread_start_us":66,"threads_started":1}
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
Jun 20 06:02:23 dist-test-slave-4h14 krb5kdc[9607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935343, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.61@KRBTEST.COM
I20260620 06:02:23.962854 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.948391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55532 (local address 127.3.115.60:40179)
0620 06:02:23.949261 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.949265 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.949280 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:23.949331 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.949336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.949393 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.950936 (+ 1543us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.951061 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.951911 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.953247 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.953584 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.954114 (+ 530us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.955770 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.955774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.955777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.955805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.957985 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.961230 (+ 3245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.961234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.961236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.961326 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.961661 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.961664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.961667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.961891 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.962025 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.962559 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.962660 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":87,"threads_started":1}
I20260620 06:02:23.963634 10905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.948857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.3.115.62:35659)
0620 06:02:23.949108 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.949123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.949380 (+ 257us) server_negotiation.cc:407] Connection header received
0620 06:02:23.949627 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.949630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.949681 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.949740 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.950841 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.951763 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.953104 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.953425 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.953498 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.956953 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.956959 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.956964 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.956993 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.959299 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.959776 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.959780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.959781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.959870 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.962448 (+ 2578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.962452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.962453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.962676 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.962808 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.963315 (+ 507us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.963445 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":83,"threads_started":1}
I20260620 06:02:23.967926 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.947169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:35167 (local address 127.3.115.62:35659)
0620 06:02:23.947300 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.947304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.949496 (+ 2192us) server_negotiation.cc:407] Connection header received
0620 06:02:23.952061 (+ 2565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.952065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.952109 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.952159 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.954193 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.954984 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.957637 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.958029 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.958130 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.963140 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.963144 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.963150 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.963178 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.965645 (+ 2467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966564 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.966567 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.966569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.966654 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966984 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.966987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.966989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.967180 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.967257 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.967681 (+ 424us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.967788 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:02:23.968828 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.948026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:40425 (local address 127.3.115.60:40179)
0620 06:02:23.948746 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.948751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.949697 (+ 946us) server_negotiation.cc:407] Connection header received
0620 06:02:23.950985 (+ 1288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.950989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.951037 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.951090 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.954239 (+ 3149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.954996 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.957654 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.958030 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.958108 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.963547 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.963550 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.963553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.963573 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.965372 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966162 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.966165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.966167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.966280 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966693 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.966698 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.966700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.966943 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.967052 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.968540 (+ 1488us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.968685 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617}
I20260620 06:02:23.969209 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.947711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:35167)
0620 06:02:23.949359 (+ 1648us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.949418 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.950555 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.952588 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.952591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.953939 (+ 1348us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.954073 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.954087 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.955135 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.955138 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.957496 (+ 2358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.957508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.958305 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.958311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.958944 (+ 633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.960027 (+ 1083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.960045 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.962777 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.966054 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966057 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.966059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.966450 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.966778 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.966783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.966878 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.967314 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.967316 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.967386 (+ 70us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.967595 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.968019 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1430,"mutex_wait_us":1209,"spinlock_wait_cycles":84736,"thread_start_us":167,"threads_started":1}
I20260620 06:02:23.969694 10902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.947711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.3.115.3:40425)
0620 06:02:23.949618 (+ 1907us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.949633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.950765 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.951850 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.951859 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.953002 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.953676 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.953696 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.956558 (+ 2862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.956565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.957514 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.957529 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.958304 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.958309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.958648 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.959202 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.959227 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.963288 (+ 4061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.965645 (+ 2357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.965651 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.965671 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.966047 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.966371 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.966383 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.966572 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.967040 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.967048 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.967386 (+ 338us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.967595 (+ 209us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.968805 (+ 1210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"mutex_wait_us":953,"spinlock_wait_cycles":8960,"thread_start_us":139,"threads_started":1}
I20260620 06:02:23.970371 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.947692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:49813 (local address 127.3.115.61:35185)
0620 06:02:23.947827 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.947831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.949952 (+ 2121us) server_negotiation.cc:407] Connection header received
0620 06:02:23.951232 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.951235 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.951281 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.951332 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:23.955001 (+ 3669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.955823 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.959319 (+ 3496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.959660 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.959749 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.963667 (+ 3918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:23.963671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:23.963674 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:23.963698 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:23.965624 (+ 1926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.969133 (+ 3509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.969137 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.969139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.969226 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:23.969502 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:23.969505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:23.969507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:23.969701 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:23.969781 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.970101 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.970223 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":3456}
I20260620 06:02:23.970834 10892 heartbeater.cc:347] Connected to a master server at 127.3.115.60:40179
I20260620 06:02:23.970875 10893 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35659
I20260620 06:02:23.967324 9652 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260620 06:02:23.971078 10893 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.971158 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.947711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.3.115.3:49813)
0620 06:02:23.949879 (+ 2168us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.949893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.950836 (+ 943us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.952284 (+ 1448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.952288 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.954705 (+ 2417us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:23.954848 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.954860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.955976 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.955979 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.959180 (+ 3201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.959192 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.959935 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.959942 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.960036 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.960599 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:23.960618 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:23.963125 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:23.966149 (+ 3024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.966153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.966156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.969006 (+ 2850us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.969327 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:23.969330 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:23.969332 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:23.969410 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:23.969835 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:23.969838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:23.969921 (+ 83us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.970541 (+ 620us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.970682 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2066,"mutex_wait_us":28,"spinlock_wait_cycles":435072,"thread_start_us":308,"threads_started":1}
I20260620 06:02:23.971077 10892 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.971659 10893 heartbeater.cc:511] Master 127.3.115.62:35659 requested a full tablet report, sending...
I20260620 06:02:23.971727 10892 heartbeater.cc:511] Master 127.3.115.60:40179 requested a full tablet report, sending...
W20260620 06:02:23.972129 9652 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:23.972395 10894 heartbeater.cc:347] Connected to a master server at 127.3.115.61:35185
I20260620 06:02:23.972463 10894 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:23.972613 10894 heartbeater.cc:511] Master 127.3.115.61:35185 requested a full tablet report, sending...
I20260620 06:02:23.972849 9651 ts_manager.cc:194] Re-registered known tserver with Master: 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649)
I20260620 06:02:23.973348 10255 ts_manager.cc:194] Registered new tserver with Master: 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649)
I20260620 06:02:23.973613 9651 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:35167
I20260620 06:02:23.974272 10437 ts_manager.cc:194] Registered new tserver with Master: 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649)
I20260620 06:02:23.975322 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: NON_VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: true } } }
I20260620 06:02:23.975445 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.979203 10762 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: attempting to promote NON_VOTER e9f073c9072d49fba384d895994c557c to VOTER
I20260620 06:02:23.979964 10762 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:23.981400 10448 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 LEARNER]: Refusing update from remote peer 7804e9288f6f4e179d4d76ffb820f536: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 06:02:23.981599 10264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Refusing update from remote peer 7804e9288f6f4e179d4d76ffb820f536: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 06:02:23.981801 10910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260620 06:02:23.982129 10910 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [LEADER]: Connected to new peer: Peer: permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 06:02:23.983824 10762 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e9f073c9072d49fba384d895994c557c (127.3.115.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.984313 10264 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e9f073c9072d49fba384d895994c557c (127.3.115.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.984462 10448 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, e9f073c9072d49fba384d895994c557c (127.3.115.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.985486 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.985586 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.985702 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.985780 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.985877 10762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.985948 10762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:23.986675 10477 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.986788 10477 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:23.988775 10920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7804e9288f6f4e179d4d76ffb820f536" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:23.988970 10920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:23.992945 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.987756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:35649 (local address 127.0.0.1:59050)
0620 06:02:23.987990 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.988004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.988077 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.989200 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.989204 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.989222 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:23.989309 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.989315 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.990553 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.990558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.991302 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.991310 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.992408 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.992412 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.992597 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.992625 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.992723 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.992796 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":24,"thread_start_us":99,"threads_started":1}
I20260620 06:02:23.993085 10763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.987546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:45025 (local address 127.0.0.1:38544)
0620 06:02:23.987653 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.987670 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.987765 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.988879 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.988882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.988903 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:23.989017 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.989026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.990427 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.990430 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.991506 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.991515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.992486 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.992490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.992789 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.992812 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.992894 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.992965 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":185}
I20260620 06:02:23.993151 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.987845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38544 (local address 127.3.115.1:45025)
0620 06:02:23.988309 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.988325 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.988350 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:02:23.988437 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.988446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.988730 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.988912 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:23.989140 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.990306 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.991630 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.992360 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.992621 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.992801 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.992925 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.993021 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":105,"threads_started":1}
I20260620 06:02:23.993767 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.988161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59050 (local address 127.3.115.3:35649)
0620 06:02:23.988755 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.988768 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.988791 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:02:23.988870 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.988875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.989079 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.989230 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:23.989405 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.990437 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.991426 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.992277 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.992408 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.992576 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.993541 (+ 965us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.993637 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":91,"threads_started":1}
I20260620 06:02:23.994995 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.988525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50866 (local address 127.3.115.2:46583)
0620 06:02:23.989280 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:23.989294 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:23.989314 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:02:23.989390 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:23.989397 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:23.989569 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:23.989724 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:23.989884 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.991553 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.992616 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.993511 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.993644 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.993793 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:23.994751 (+ 958us) server_negotiation.cc:299] Negotiation successful
0620 06:02:23.994855 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":654,"thread_start_us":130,"threads_started":1}
I20260620 06:02:23.995108 10923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:23.988172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:46583 (local address 127.0.0.1:50866)
0620 06:02:23.988403 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:23.988416 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:23.988493 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:23.989699 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:23.989702 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:23.989718 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:23.989794 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.989799 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.991698 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.991703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.992476 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:23.992484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:23.993645 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:23.993649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:23.993731 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:23.993750 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:02:23.993843 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:02:23.994927 (+ 1084us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":90,"threads_started":1}
I20260620 06:02:23.996371 10826 tablet_service.cc:1467] Processing CreateTablet for tablet 58f4b35452d544d59d57bb4c80ca5ff8 (DEFAULT_TABLE table=test-table [id=2a288b66533d4718a7da4556182aeda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:23.996371 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 58f4b35452d544d59d57bb4c80ca5ff8 (DEFAULT_TABLE table=test-table [id=2a288b66533d4718a7da4556182aeda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:23.997520 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f4b35452d544d59d57bb4c80ca5ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:23.997520 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f4b35452d544d59d57bb4c80ca5ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:23.997659 10688 tablet_service.cc:1467] Processing CreateTablet for tablet 58f4b35452d544d59d57bb4c80ca5ff8 (DEFAULT_TABLE table=test-table [id=2a288b66533d4718a7da4556182aeda6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:23.998564 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58f4b35452d544d59d57bb4c80ca5ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:24.005146 10926 tablet_bootstrap.cc:492] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Bootstrap starting.
I20260620 06:02:24.006600 10927 tablet_bootstrap.cc:492] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Bootstrap starting.
I20260620 06:02:24.006676 10928 tablet_bootstrap.cc:492] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Bootstrap starting.
I20260620 06:02:24.007727 10926 tablet_bootstrap.cc:654] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:24.008452 10926 log.cc:824] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:24.008714 10927 tablet_bootstrap.cc:654] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:24.008713 10928 tablet_bootstrap.cc:654] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:24.009527 10927 log.cc:824] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:24.009527 10928 log.cc:824] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:24.010471 10926 tablet_bootstrap.cc:492] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: No bootstrap required, opened a new log
I20260620 06:02:24.010744 10926 ts_tablet_manager.cc:1397] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:02:24.011445 10928 tablet_bootstrap.cc:492] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: No bootstrap required, opened a new log
I20260620 06:02:24.011445 10927 tablet_bootstrap.cc:492] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: No bootstrap required, opened a new log
I20260620 06:02:24.011615 10927 ts_tablet_manager.cc:1397] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260620 06:02:24.011615 10928 ts_tablet_manager.cc:1397] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:02:24.014647 10926 raft_consensus.cc:348] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.014891 10926 raft_consensus.cc:374] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:24.014951 10926 raft_consensus.cc:729] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09430c33e5d42eebc0403da38b85de9, State: Initialized, Role: FOLLOWER
I20260620 06:02:24.015404 10926 consensus_queue.cc:260] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.016256 10926 ts_tablet_manager.cc:1428] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:02:24.016112 10927 raft_consensus.cc:348] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.016112 10928 raft_consensus.cc:348] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.016348 10928 raft_consensus.cc:374] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:24.016348 10927 raft_consensus.cc:374] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:24.016417 10928 raft_consensus.cc:729] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22662410bfb74fa091248719a263f65f, State: Initialized, Role: FOLLOWER
I20260620 06:02:24.016423 10927 raft_consensus.cc:729] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a41d09bee54c1194739c2405386a2c, State: Initialized, Role: FOLLOWER
I20260620 06:02:24.016687 10755 heartbeater.cc:503] Master 127.3.115.62:35659 was elected leader, sending a full tablet report...
I20260620 06:02:24.016904 10927 consensus_queue.cc:260] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.016904 10928 consensus_queue.cc:260] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.017812 10928 ts_tablet_manager.cc:1428] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:02:24.017812 10927 ts_tablet_manager.cc:1428] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 06:02:24.018328 10893 heartbeater.cc:503] Master 127.3.115.62:35659 was elected leader, sending a full tablet report...
I20260620 06:02:24.020787 10618 heartbeater.cc:503] Master 127.3.115.62:35659 was elected leader, sending a full tablet report...
I20260620 06:02:24.021912 10932 raft_consensus.cc:482] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:24.022053 10932 raft_consensus.cc:504] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
W20260620 06:02:24.023505 10620 tablet.cc:2369] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:24.025007 10933 raft_consensus.cc:482] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:24.025159 10933 raft_consensus.cc:504] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.026268 10933 leader_election.cc:288] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22662410bfb74fa091248719a263f65f (127.3.115.1:45025), d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583)
I20260620 06:02:24.026268 10932 leader_election.cc:288] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22662410bfb74fa091248719a263f65f (127.3.115.1:45025), 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649)
I20260620 06:02:24.031637 10758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.025958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:45025 (local address 127.3.115.2:45377)
0620 06:02:24.026083 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:24.026100 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:24.026180 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:24.028037 (+ 1857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:24.028039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:24.028056 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:24.028140 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.028146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.029128 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.029131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.030256 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.030267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.031139 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.031143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.031282 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.031311 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:02:24.031381 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:02:24.031462 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 06:02:24.031752 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.027135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:46583 (local address 127.3.115.3:35089)
0620 06:02:24.027244 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:24.027259 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:24.027347 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:24.027665 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:24.027671 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:24.027711 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:24.027824 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.027832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.028769 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.028773 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.030472 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.030481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.031335 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.031339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.031440 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.031460 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:02:24.031531 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:02:24.031611 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":440}
I20260620 06:02:24.031795 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.025967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:35649 (local address 127.3.115.2:49239)
0620 06:02:24.026410 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:24.026424 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:24.026538 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:24.026854 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:24.026859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:24.026887 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:24.027003 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.027013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.027985 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.027991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.029253 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.029261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.031409 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.031415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.031527 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.031546 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:02:24.031605 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 06:02:24.031667 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":194}
I20260620 06:02:24.032121 10922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.026181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:45377 (local address 127.3.115.1:45025)
0620 06:02:24.026695 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.026701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.026717 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:02:24.026758 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.026761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.026800 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.026865 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:24.028268 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.029008 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.030386 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.031012 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.031112 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.031174 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.031908 (+ 734us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.032007 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433}
I20260620 06:02:24.032176 10925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.027374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:35089 (local address 127.3.115.2:46583)
0620 06:02:24.027466 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.027470 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.027481 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:02:24.027516 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.027519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.027553 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.027626 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:24.027950 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.028622 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.030593 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.031210 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.031314 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.031366 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.032009 (+ 643us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.032079 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:02:24.032238 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.026120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:49239 (local address 127.3.115.3:35649)
0620 06:02:24.026319 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.026323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.026500 (+ 177us) server_negotiation.cc:407] Connection header received
0620 06:02:24.026618 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.026622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.026667 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.026735 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:24.027128 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.027863 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.029376 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.030049 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.030140 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.030197 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.032041 (+ 1844us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.032125 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":107}
I20260620 06:02:24.032703 10708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09430c33e5d42eebc0403da38b85de9" is_pre_election: true
I20260620 06:02:24.032692 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "d09430c33e5d42eebc0403da38b85de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a41d09bee54c1194739c2405386a2c" is_pre_election: true
I20260620 06:02:24.032711 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "d09430c33e5d42eebc0403da38b85de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22662410bfb74fa091248719a263f65f" is_pre_election: true
I20260620 06:02:24.032922 10708 raft_consensus.cc:2454] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 0.
I20260620 06:02:24.032922 10846 raft_consensus.cc:2454] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09430c33e5d42eebc0403da38b85de9 in term 0.
I20260620 06:02:24.033015 10571 raft_consensus.cc:2454] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d09430c33e5d42eebc0403da38b85de9 in term 0.
I20260620 06:02:24.033351 10779 leader_election.cc:302] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a41d09bee54c1194739c2405386a2c, d09430c33e5d42eebc0403da38b85de9; no voters:
I20260620 06:02:24.033351 10644 leader_election.cc:302] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69a41d09bee54c1194739c2405386a2c, d09430c33e5d42eebc0403da38b85de9; no voters:
I20260620 06:02:24.033521 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.027864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:45025 (local address 127.3.115.3:57821)
0620 06:02:24.027998 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:24.028012 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:24.028083 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:24.029069 (+ 986us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:24.029072 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:24.029088 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:24.029187 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.029193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.030237 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.030241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.031162 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:24.031177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.033119 (+ 1942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.033123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:24.033246 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.033275 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:02:24.033331 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 06:02:24.033402 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 06:02:24.033830 10932 raft_consensus.cc:2781] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:24.033936 10932 raft_consensus.cc:482] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:24.033996 10932 raft_consensus.cc:3037] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:24.034137 10936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.028387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:57821 (local address 127.3.115.1:45025)
0620 06:02:24.028845 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.028853 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.028864 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:02:24.028900 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.028902 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.028942 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.029289 (+ 347us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:24.029386 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.030110 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.031323 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.032799 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.033829 (+ 1030us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.033887 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.033915 (+ 28us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.033994 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":110,"threads_started":1}
I20260620 06:02:24.034350 10933 raft_consensus.cc:2781] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:24.034442 10933 raft_consensus.cc:482] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:24.034494 10933 raft_consensus.cc:3037] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:24.034498 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22662410bfb74fa091248719a263f65f" is_pre_election: true
I20260620 06:02:24.034626 10571 raft_consensus.cc:2454] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 0.
I20260620 06:02:24.035372 10932 raft_consensus.cc:504] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.035557 10933 raft_consensus.cc:504] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.035775 10932 leader_election.cc:288] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [CANDIDATE]: Term 1 election: Requested vote from peers 22662410bfb74fa091248719a263f65f (127.3.115.1:45025), 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649)
I20260620 06:02:24.036135 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22662410bfb74fa091248719a263f65f"
I20260620 06:02:24.036155 10846 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "d09430c33e5d42eebc0403da38b85de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a41d09bee54c1194739c2405386a2c"
I20260620 06:02:24.036244 10571 raft_consensus.cc:3037] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:24.036311 10933 leader_election.cc:288] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 election: Requested vote from peers 22662410bfb74fa091248719a263f65f (127.3.115.1:45025), d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583)
I20260620 06:02:24.036332 10846 raft_consensus.cc:2379] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d09430c33e5d42eebc0403da38b85de9 in current term 1: Already voted for candidate 69a41d09bee54c1194739c2405386a2c in this term.
I20260620 06:02:24.036386 10708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09430c33e5d42eebc0403da38b85de9"
I20260620 06:02:24.036546 10708 raft_consensus.cc:2379] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 69a41d09bee54c1194739c2405386a2c in current term 1: Already voted for candidate d09430c33e5d42eebc0403da38b85de9 in this term.
I20260620 06:02:24.036772 10570 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58f4b35452d544d59d57bb4c80ca5ff8" candidate_uuid: "d09430c33e5d42eebc0403da38b85de9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22662410bfb74fa091248719a263f65f"
I20260620 06:02:24.037256 10643 leader_election.cc:302] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d09430c33e5d42eebc0403da38b85de9; no voters: 22662410bfb74fa091248719a263f65f, 69a41d09bee54c1194739c2405386a2c
I20260620 06:02:24.037452 10932 raft_consensus.cc:2726] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 06:02:24.037613 10571 raft_consensus.cc:2454] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 1.
I20260620 06:02:24.037906 10781 leader_election.cc:302] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 22662410bfb74fa091248719a263f65f, 69a41d09bee54c1194739c2405386a2c; no voters: d09430c33e5d42eebc0403da38b85de9
I20260620 06:02:24.038058 10933 raft_consensus.cc:2781] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:24.038268 10933 raft_consensus.cc:686] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 1 LEADER]: Becoming Leader. State: Replica: 69a41d09bee54c1194739c2405386a2c, State: Running, Role: LEADER
I20260620 06:02:24.038617 10933 consensus_queue.cc:237] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } }
I20260620 06:02:24.043066 9651 catalog_manager.cc:5495] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a41d09bee54c1194739c2405386a2c (127.3.115.3). New cstate: current_term: 1 leader_uuid: "69a41d09bee54c1194739c2405386a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } health_report { overall_health: UNKNOWN } } }
I20260620 06:02:24.061108 10924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.057756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.3.115.3:35649)
0620 06:02:24.057874 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.057882 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.057897 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:24.058053 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.058055 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.058146 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.058233 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:24.058409 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.058890 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.059672 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.059905 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.059990 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.060262 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:24.060739 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:24.060846 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.060893 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.060939 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 06:02:24.066973 10571 raft_consensus.cc:1261] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Refusing update from remote peer 69a41d09bee54c1194739c2405386a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:24.066977 10708 raft_consensus.cc:1261] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Refusing update from remote peer 69a41d09bee54c1194739c2405386a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:24.067584 10933 consensus_queue.cc:1035] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:24.067873 10937 consensus_queue.cc:1035] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:24.076884 10942 mvcc.cc:205] Tried to move back new op lower bound from 7298807169291632640 to 7298807169185103872. Current Snapshot: MvccSnapshot[applied={T|T < 7298807169291632640}]
I20260620 06:02:24.086434 9651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:24.086623 9651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57040:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:24.091228 9651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7804e9288f6f4e179d4d76ffb820f536: Sending DeleteTablet for 3 replicas of tablet 58f4b35452d544d59d57bb4c80ca5ff8
I20260620 06:02:24.092005 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f4b35452d544d59d57bb4c80ca5ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:24 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:59050
I20260620 06:02:24.092310 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f4b35452d544d59d57bb4c80ca5ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:24 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:50866
I20260620 06:02:24.092525 10950 tablet_replica.cc:330] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: stopping tablet replica
I20260620 06:02:24.092885 10950 raft_consensus.cc:2229] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:24.093056 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 58f4b35452d544d59d57bb4c80ca5ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:24 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:38544
I20260620 06:02:24.093209 10950 raft_consensus.cc:2258] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:24.093597 10951 tablet_replica.cc:330] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: stopping tablet replica
I20260620 06:02:24.093624 10953 tablet_replica.cc:330] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: stopping tablet replica
I20260620 06:02:24.093917 10953 raft_consensus.cc:2229] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:24.094235 10953 raft_consensus.cc:2258] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:24.095341 10950 ts_tablet_manager.cc:1905] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:24.095722 10953 ts_tablet_manager.cc:1905] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:24.096104 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 9620
I20260620 06:02:24.099079 10950 ts_tablet_manager.cc:1918] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:24.099182 10950 log.cc:1196] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/58f4b35452d544d59d57bb4c80ca5ff8
I20260620 06:02:24.099354 10953 ts_tablet_manager.cc:1918] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:24.099443 10953 log.cc:1196] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/58f4b35452d544d59d57bb4c80ca5ff8
I20260620 06:02:24.099516 10950 ts_tablet_manager.cc:1939] T 58f4b35452d544d59d57bb4c80ca5ff8 P 69a41d09bee54c1194739c2405386a2c: Deleting consensus metadata
I20260620 06:02:24.099768 10953 ts_tablet_manager.cc:1939] T 58f4b35452d544d59d57bb4c80ca5ff8 P 22662410bfb74fa091248719a263f65f: Deleting consensus metadata
I20260620 06:02:24.100891 10951 raft_consensus.cc:2229] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:24.101238 10951 raft_consensus.cc:2258] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:24.102773 10951 ts_tablet_manager.cc:1905] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:02:24.103897 10780 connection.cc:535] server connection from 127.0.0.1:59050 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 06:02:24.103916 10507 connection.cc:535] server connection from 127.0.0.1:38544 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:02:24.105727 10951 ts_tablet_manager.cc:1918] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:24.105831 10951 log.cc:1196] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/58f4b35452d544d59d57bb4c80ca5ff8
I20260620 06:02:24.106133 10951 ts_tablet_manager.cc:1939] T 58f4b35452d544d59d57bb4c80ca5ff8 P d09430c33e5d42eebc0403da38b85de9: Deleting consensus metadata
W20260620 06:02:24.107205 10644 connection.cc:441] server connection from 127.0.0.1:50866 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50866 (request call id 1) could send its response
I20260620 06:02:24.115331 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.094094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.3.115.61:35185)
0620 06:02:24.094319 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.094324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.105471 (+ 11147us) server_negotiation.cc:407] Connection header received
0620 06:02:24.105672 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.105675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.105728 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.105782 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:24.106791 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.107525 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.108367 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.108731 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.108845 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.111138 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:24.111142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:24.111145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:24.111190 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:24.113008 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.113761 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.113765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.113767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.113852 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.114142 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.114146 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.114148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.114599 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:24.114704 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.115054 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.115167 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260620 06:02:24.119712 10907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.109052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.3.115.61:35185)
0620 06:02:24.109156 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.109161 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.109407 (+ 246us) server_negotiation.cc:407] Connection header received
0620 06:02:24.109606 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.109610 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.109660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.109714 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:24.110666 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.111447 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.112548 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.112925 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.113000 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.115056 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:24.115060 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:24.115062 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:24.115087 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:24.117366 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.118246 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.118251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.118255 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.118346 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.118687 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.118691 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.118692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.118905 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:24.119013 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.119399 (+ 386us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.119520 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 06:02:24.120745 10904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.109695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55550 (local address 127.3.115.60:40179)
0620 06:02:24.109836 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.109841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.110289 (+ 448us) server_negotiation.cc:407] Connection header received
0620 06:02:24.110473 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.110477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.110531 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.110588 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:24.111505 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.112335 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.113547 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.113789 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.113908 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.115370 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:24.115374 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:24.115379 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:24.115416 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:24.118950 (+ 3534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.119401 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.119405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.119407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.119489 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.119781 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.119785 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.119787 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.120005 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:24.120114 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.120455 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.120563 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:02:24.124343 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:24.094188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.3.115.60:40179)
0620 06:02:24.094407 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:24.094412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:24.115904 (+ 21492us) server_negotiation.cc:407] Connection header received
0620 06:02:24.115987 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:24.115991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:24.116036 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:24.116117 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:24.116933 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.117725 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.118841 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:24.119173 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:24.119240 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:24.120787 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:24.120790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:24.120793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:24.120816 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:24.122647 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.123100 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.123102 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.123104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.123172 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:24.123435 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:24.123437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:24.123438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:24.123588 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:24.123679 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:24.123970 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 06:02:24.124187 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260620 06:02:25.079726 10961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.079188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:34027)
0620 06:02:25.079505 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.079601 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":74,"threads_started":1}
I20260620 06:02:25.079738 10960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.079083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:51359)
0620 06:02:25.079510 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.079609 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":106,"threads_started":1}
I20260620 06:02:25.080022 10962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.079552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:41145)
0620 06:02:25.079831 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.079935 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":58,"threads_started":1}
W20260620 06:02:25.080308 10618 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:35659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
W20260620 06:02:25.080322 10755 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:35659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
W20260620 06:02:25.080914 10893 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:35659 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.591641 10963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7804e9288f6f4e179d4d76ffb820f536)
I20260620 06:02:25.591833 10964 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 7804e9288f6f4e179d4d76ffb820f536)
I20260620 06:02:25.591866 10963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:25.592061 10964 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:25.592892 10963 leader_election.cc:288] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659), e9f073c9072d49fba384d895994c557c (127.3.115.60:40179)
I20260620 06:02:25.593035 10964 leader_election.cc:288] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659), 051b1fc1c0a54686902499589658fb1d (127.3.115.61:35185)
I20260620 06:02:25.593792 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.593175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57576)
0620 06:02:25.593561 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.593688 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":50,"threads_started":1}
W20260620 06:02:25.593972 10423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.594009 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.593491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57560)
0620 06:02:25.593851 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.593919 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":113,"threads_started":1}
W20260620 06:02:25.594230 10240 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.594905 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.594584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57580)
0620 06:02:25.594780 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.594837 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260620 06:02:25.595008 10423 leader_election.cc:334] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.595126 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.594882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57584)
0620 06:02:25.595018 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.595070 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260620 06:02:25.595256 10240 leader_election.cc:334] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.597711 10969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.593462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55566 (local address 127.3.115.60:40179)
0620 06:02:25.593935 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:25.593940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:25.593952 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:02:25.593998 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:25.594001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:25.594050 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:25.594136 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:25.594468 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.595317 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.596630 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.597252 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.597361 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:25.597425 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:25.597570 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 06:02:25.597627 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":67,"threads_started":1}
I20260620 06:02:25.597741 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.593110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40179 (local address 127.0.0.1:55566)
0620 06:02:25.593468 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.593483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:25.593658 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:25.594156 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:25.594159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:25.594176 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:25.594297 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:25.594305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.595426 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.595429 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:25.596520 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:25.596527 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.597363 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.597366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:25.597433 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:25.597453 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:02:25.597548 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:02:25.597620 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":59,"threads_started":1}
I20260620 06:02:25.598225 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "051b1fc1c0a54686902499589658fb1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e9f073c9072d49fba384d895994c557c" is_pre_election: true
I20260620 06:02:25.598498 10448 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 051b1fc1c0a54686902499589658fb1d in term 1.
I20260620 06:02:25.598722 10968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.593331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60338 (local address 127.3.115.61:35185)
0620 06:02:25.593812 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:25.593817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:25.594371 (+ 554us) server_negotiation.cc:407] Connection header received
0620 06:02:25.594776 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:25.594779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:25.594824 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:25.594895 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:25.595384 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.596159 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.597528 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.598130 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.598261 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:25.598314 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:25.598514 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 06:02:25.598602 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":63,"threads_started":1}
I20260620 06:02:25.598723 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.593206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:35185 (local address 127.0.0.1:60338)
0620 06:02:25.593812 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.593828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:25.594021 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:25.595060 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:25.595063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:25.595120 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:25.595257 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:25.595264 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.596291 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.596293 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:25.597419 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:25.597427 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:25.598262 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:25.598265 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:25.598360 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:25.598387 (+ 27us) client_negotiation.cc:769] Sending connection context
0620 06:02:25.598481 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 06:02:25.598566 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":67,"threads_started":1}
I20260620 06:02:25.598954 10241 leader_election.cc:302] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 051b1fc1c0a54686902499589658fb1d, e9f073c9072d49fba384d895994c557c; no voters: 7804e9288f6f4e179d4d76ffb820f536
I20260620 06:02:25.599102 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f073c9072d49fba384d895994c557c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "051b1fc1c0a54686902499589658fb1d" is_pre_election: true
I20260620 06:02:25.599176 10963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:02:25.599290 10264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f073c9072d49fba384d895994c557c in term 1.
I20260620 06:02:25.599434 10963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7804e9288f6f4e179d4d76ffb820f536)
I20260620 06:02:25.599514 10963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:02:25.599645 10422 leader_election.cc:302] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 051b1fc1c0a54686902499589658fb1d, e9f073c9072d49fba384d895994c557c; no voters: 7804e9288f6f4e179d4d76ffb820f536
I20260620 06:02:25.600009 10964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:02:25.600069 10964 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Starting leader election (detected failure of leader 7804e9288f6f4e179d4d76ffb820f536)
I20260620 06:02:25.600107 10964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:02:25.600901 10963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:25.601321 10963 leader_election.cc:288] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [CANDIDATE]: Term 2 election: Requested vote from peers 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659), e9f073c9072d49fba384d895994c557c (127.3.115.60:40179)
I20260620 06:02:25.601420 10964 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:25.601656 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.601393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57588)
0620 06:02:25.601527 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.601584 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 06:02:25.601648 10448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "051b1fc1c0a54686902499589658fb1d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e9f073c9072d49fba384d895994c557c"
I20260620 06:02:25.601810 10448 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 051b1fc1c0a54686902499589658fb1d in current term 2: Already voted for candidate e9f073c9072d49fba384d895994c557c in this term.
I20260620 06:02:25.601810 10964 leader_election.cc:288] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 2 election: Requested vote from peers 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659), 051b1fc1c0a54686902499589658fb1d (127.3.115.61:35185)
I20260620 06:02:25.602138 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f073c9072d49fba384d895994c557c" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "051b1fc1c0a54686902499589658fb1d"
I20260620 06:02:25.602295 10264 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e9f073c9072d49fba384d895994c557c in current term 2: Already voted for candidate 051b1fc1c0a54686902499589658fb1d in this term.
I20260620 06:02:25.602417 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.602148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57604)
0620 06:02:25.602311 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.602363 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260620 06:02:25.602501 10240 leader_election.cc:334] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.602608 10240 leader_election.cc:302] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 051b1fc1c0a54686902499589658fb1d; no voters: 7804e9288f6f4e179d4d76ffb820f536, e9f073c9072d49fba384d895994c557c
I20260620 06:02:25.602746 10963 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 06:02:25.602861 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.601953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57594)
0620 06:02:25.602371 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.602427 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344}
I20260620 06:02:25.603479 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:25.603288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57620)
0620 06:02:25.603387 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:25.603433 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:02:25.603585 10423 leader_election.cc:334] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:25.603659 10423 leader_election.cc:302] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9f073c9072d49fba384d895994c557c; no voters: 051b1fc1c0a54686902499589658fb1d, 7804e9288f6f4e179d4d76ffb820f536
I20260620 06:02:25.603785 10964 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260620 06:02:26.081465 10974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:26.080918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:52287)
0620 06:02:26.081237 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:26.081349 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":69,"threads_started":1}
I20260620 06:02:26.082137 10973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:26.080891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:45767)
0620 06:02:26.081978 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:26.082038 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":112,"threads_started":1}
I20260620 06:02:26.082782 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:26.081423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:50445)
0620 06:02:26.082063 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:26.082656 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":82,"threads_started":1}
I20260620 06:02:27.083010 10976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.082453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:40633)
0620 06:02:27.082812 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.082882 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
I20260620 06:02:27.084165 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.083613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:45095)
0620 06:02:27.083986 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.084050 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
I20260620 06:02:27.084188 10977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.082847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:45583)
0620 06:02:27.084032 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.084104 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1113,"thread_start_us":67,"threads_started":1}
I20260620 06:02:27.517617 10979 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:27.517895 10979 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:27.518399 10979 leader_election.cc:288] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659), 051b1fc1c0a54686902499589658fb1d (127.3.115.61:35185)
I20260620 06:02:27.518914 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f073c9072d49fba384d895994c557c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "051b1fc1c0a54686902499589658fb1d" is_pre_election: true
I20260620 06:02:27.519088 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.518602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57890)
0620 06:02:27.518902 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.518958 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":61,"threads_started":1}
I20260620 06:02:27.519060 10264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9f073c9072d49fba384d895994c557c in term 2.
I20260620 06:02:27.519502 10422 leader_election.cc:302] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051b1fc1c0a54686902499589658fb1d, e9f073c9072d49fba384d895994c557c; no voters:
I20260620 06:02:27.519727 10979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260620 06:02:27.519789 10979 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:27.519834 10979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:02:27.519928 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.519691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57898)
0620 06:02:27.519795 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.519858 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 06:02:27.520038 10423 leader_election.cc:334] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:27.520973 10979 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:27.521296 10979 leader_election.cc:288] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 3 election: Requested vote from peers 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659), 051b1fc1c0a54686902499589658fb1d (127.3.115.61:35185)
I20260620 06:02:27.521606 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.521370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57910)
0620 06:02:27.521485 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.521534 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:02:27.521687 10264 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9f073c9072d49fba384d895994c557c" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "051b1fc1c0a54686902499589658fb1d"
I20260620 06:02:27.521831 10264 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:02:27.523003 10264 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9f073c9072d49fba384d895994c557c in term 3.
I20260620 06:02:27.523309 10422 leader_election.cc:302] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 051b1fc1c0a54686902499589658fb1d, e9f073c9072d49fba384d895994c557c; no voters:
I20260620 06:02:27.523464 10979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:02:27.523666 10979 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [term 3 LEADER]: Becoming Leader. State: Replica: e9f073c9072d49fba384d895994c557c, State: Running, Role: LEADER
I20260620 06:02:27.523835 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.523603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57912)
0620 06:02:27.523713 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.523767 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:02:27.524044 10423 leader_election.cc:334] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
I20260620 06:02:27.524004 10979 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } }
I20260620 06:02:27.525413 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f073c9072d49fba384d895994c557c. Latest consensus state: current_term: 3 leader_uuid: "e9f073c9072d49fba384d895994c557c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:27.525539 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:27.525841 10984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:27.527983 10984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:27.528347 10984 catalog_manager.cc:1259] Loaded cluster ID: ab969c50d19740719686c797238742e6
I20260620 06:02:27.528391 10984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:27.528961 10984 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:27.529317 10984 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Loaded TSK: 0
I20260620 06:02:27.529954 10984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:27.576797 10438 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:27.580225 10438 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:27.583590 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.583279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57928)
0620 06:02:27.583426 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.583501 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 06:02:27.583662 10264 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [term 3 FOLLOWER]: Refusing update from remote peer e9f073c9072d49fba384d895994c557c: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260620 06:02:27.584249 10979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [LEADER]: Connected to new peer: Peer: permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260620 06:02:27.584367 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.584075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.0.0.1:57936)
0620 06:02:27.584228 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.584274 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:02:27.584542 10423 consensus_peers.cc:487] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c -> Peer 7804e9288f6f4e179d4d76ffb820f536 (127.3.115.62:35659): Couldn't send request to peer 7804e9288f6f4e179d4d76ffb820f536. Status: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 06:02:27.585999 10986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9f073c9072d49fba384d895994c557c. Latest consensus state: current_term: 3 leader_uuid: "e9f073c9072d49fba384d895994c557c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:27.586158 10986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:27.587427 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e9f073c9072d49fba384d895994c557c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:27.587549 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:27.588178 10986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e9f073c9072d49fba384d895994c557c" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7804e9288f6f4e179d4d76ffb820f536" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35659 } } peers { permanent_uuid: "051b1fc1c0a54686902499589658fb1d" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 35185 } attrs { promote: false } } peers { permanent_uuid: "e9f073c9072d49fba384d895994c557c" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40179 } attrs { promote: false } } }
I20260620 06:02:27.588356 10986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 051b1fc1c0a54686902499589658fb1d [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:02:27.598194 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.593482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:45025 (local address 127.0.0.1:38548)
0620 06:02:27.593742 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.593756 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:27.593841 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:27.594702 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:27.594705 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:27.594720 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:27.594832 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:27.594838 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.595876 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.595880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:27.596950 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:27.596959 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.597797 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.597799 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:27.597864 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.597883 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:02:27.597938 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 06:02:27.597993 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":112,"thread_start_us":88,"threads_started":1}
I20260620 06:02:27.598183 10993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.593825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.3.115.1:45025)
0620 06:02:27.594481 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:27.594485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:27.594499 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:27.594555 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:27.594557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:27.594597 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:27.594694 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:27.594945 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.595762 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.597079 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.597685 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.597801 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.597867 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:27.597967 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 06:02:27.598037 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":80,"threads_started":1}
I20260620 06:02:27.598701 10551 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3f9ad7a976474489e0a29c9aad0e72 (DEFAULT_TABLE table=test-table [id=36a6f0b50e214242b9d15d3e435b39b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:27.599057 10551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3f9ad7a976474489e0a29c9aad0e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:27.599417 10994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.593887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:35649 (local address 127.0.0.1:59064)
0620 06:02:27.594220 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.594234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:27.594316 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:27.594769 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:27.594771 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:27.594789 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:27.594890 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:27.594899 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.595951 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.595975 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:27.597371 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:27.597381 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.598909 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.598913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:27.599020 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.599057 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:02:27.599130 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:02:27.599258 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":60,"thread_start_us":85,"threads_started":1}
I20260620 06:02:27.599553 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.593200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:46583 (local address 127.0.0.1:50878)
0620 06:02:27.593325 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.593342 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:27.593439 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:27.595154 (+ 1715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:27.595156 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:27.595171 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:27.595245 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:27.595250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.596009 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.596012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:27.597374 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:27.597381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.599078 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.599084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:27.599261 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.599281 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:02:27.599357 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:02:27.599418 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":161}
I20260620 06:02:27.599717 10991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.593357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50878 (local address 127.3.115.2:46583)
0620 06:02:27.594934 (+ 1577us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:27.594938 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:27.594950 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:02:27.595003 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:27.595005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:27.595042 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:27.595133 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:27.595362 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.595903 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.597477 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.598966 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.599062 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.599129 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:27.599392 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 06:02:27.599474 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1480,"thread_start_us":79,"threads_started":1}
I20260620 06:02:27.600030 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.594231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59064 (local address 127.3.115.3:35649)
0620 06:02:27.594486 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:27.594493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:27.594508 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:27.594558 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:27.594561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:27.594612 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:27.594763 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:27.595031 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.595827 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.598134 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.598733 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.598865 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.598931 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:27.599772 (+ 841us) server_negotiation.cc:299] Negotiation successful
0620 06:02:27.599900 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":112,"threads_started":1}
I20260620 06:02:27.600485 10826 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3f9ad7a976474489e0a29c9aad0e72 (DEFAULT_TABLE table=test-table [id=36a6f0b50e214242b9d15d3e435b39b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:27.600790 10826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3f9ad7a976474489e0a29c9aad0e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:27.600946 10688 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3f9ad7a976474489e0a29c9aad0e72 (DEFAULT_TABLE table=test-table [id=36a6f0b50e214242b9d15d3e435b39b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:27.601251 10688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3f9ad7a976474489e0a29c9aad0e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:27.601598 10996 tablet_bootstrap.cc:492] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f: Bootstrap starting.
I20260620 06:02:27.602455 10996 tablet_bootstrap.cc:654] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:27.603469 10998 tablet_bootstrap.cc:492] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Bootstrap starting.
I20260620 06:02:27.603667 10996 tablet_bootstrap.cc:492] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f: No bootstrap required, opened a new log
I20260620 06:02:27.603741 10996 ts_tablet_manager.cc:1397] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:27.603886 10999 tablet_bootstrap.cc:492] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Bootstrap starting.
I20260620 06:02:27.604415 10998 tablet_bootstrap.cc:654] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:27.604938 10999 tablet_bootstrap.cc:654] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:27.605101 10996 raft_consensus.cc:348] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.605232 10996 raft_consensus.cc:374] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:27.605319 10996 raft_consensus.cc:729] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22662410bfb74fa091248719a263f65f, State: Initialized, Role: FOLLOWER
I20260620 06:02:27.605516 10996 consensus_queue.cc:260] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.605921 10996 ts_tablet_manager.cc:1428] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:27.606042 10998 tablet_bootstrap.cc:492] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: No bootstrap required, opened a new log
I20260620 06:02:27.606129 10998 ts_tablet_manager.cc:1397] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:27.606163 10999 tablet_bootstrap.cc:492] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: No bootstrap required, opened a new log
I20260620 06:02:27.606243 10999 ts_tablet_manager.cc:1397] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 06:02:27.606519 10998 raft_consensus.cc:348] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.606643 10998 raft_consensus.cc:374] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:27.606678 10998 raft_consensus.cc:729] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a41d09bee54c1194739c2405386a2c, State: Initialized, Role: FOLLOWER
I20260620 06:02:27.606690 10999 raft_consensus.cc:348] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.606801 10999 raft_consensus.cc:374] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:27.606788 10998 consensus_queue.cc:260] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.606835 10999 raft_consensus.cc:729] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09430c33e5d42eebc0403da38b85de9, State: Initialized, Role: FOLLOWER
I20260620 06:02:27.606990 10999 consensus_queue.cc:260] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.607071 10998 ts_tablet_manager.cc:1428] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:27.607302 10999 ts_tablet_manager.cc:1428] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:27.609055 10754 heartbeater.cc:503] Master 127.3.115.60:40179 was elected leader, sending a full tablet report...
I20260620 06:02:27.609344 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.608919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:36893)
0620 06:02:27.609162 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.609233 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":98,"threads_started":1}
I20260620 06:02:27.610713 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.609265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:49855)
0620 06:02:27.610596 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.610642 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1277,"thread_start_us":98,"threads_started":1}
I20260620 06:02:27.611140 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.606375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:43607)
0620 06:02:27.610962 (+ 4587us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.611016 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4500,"thread_start_us":85,"threads_started":1}
I20260620 06:02:27.611482 10892 heartbeater.cc:503] Master 127.3.115.60:40179 was elected leader, sending a full tablet report...
I20260620 06:02:27.612043 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.611833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:58093)
0620 06:02:27.611935 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.611992 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 06:02:27.612694 10617 heartbeater.cc:503] Master 127.3.115.60:40179 was elected leader, sending a full tablet report...
I20260620 06:02:27.613318 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.612853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:44255)
0620 06:02:27.613179 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.613248 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260620 06:02:27.626195 11004 raft_consensus.cc:482] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:27.626320 11004 raft_consensus.cc:504] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.626788 11004 leader_election.cc:288] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583), 22662410bfb74fa091248719a263f65f (127.3.115.1:45025)
I20260620 06:02:27.627276 10708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3f9ad7a976474489e0a29c9aad0e72" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09430c33e5d42eebc0403da38b85de9" is_pre_election: true
I20260620 06:02:27.627274 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3f9ad7a976474489e0a29c9aad0e72" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22662410bfb74fa091248719a263f65f" is_pre_election: true
I20260620 06:02:27.627449 10708 raft_consensus.cc:2454] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 0.
I20260620 06:02:27.627456 10571 raft_consensus.cc:2454] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 0.
I20260620 06:02:27.627784 10781 leader_election.cc:302] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 22662410bfb74fa091248719a263f65f, 69a41d09bee54c1194739c2405386a2c; no voters:
I20260620 06:02:27.627990 11004 raft_consensus.cc:2781] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:27.628058 11004 raft_consensus.cc:482] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:27.628087 11004 raft_consensus.cc:3037] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:27.629139 11004 raft_consensus.cc:504] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.629467 11004 leader_election.cc:288] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 election: Requested vote from peers d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583), 22662410bfb74fa091248719a263f65f (127.3.115.1:45025)
I20260620 06:02:27.629822 10571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3f9ad7a976474489e0a29c9aad0e72" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22662410bfb74fa091248719a263f65f"
I20260620 06:02:27.629822 10708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e3f9ad7a976474489e0a29c9aad0e72" candidate_uuid: "69a41d09bee54c1194739c2405386a2c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09430c33e5d42eebc0403da38b85de9"
I20260620 06:02:27.629957 10708 raft_consensus.cc:3037] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:27.629952 10571 raft_consensus.cc:3037] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:27.631058 10571 raft_consensus.cc:2454] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 1.
I20260620 06:02:27.631058 10708 raft_consensus.cc:2454] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69a41d09bee54c1194739c2405386a2c in term 1.
I20260620 06:02:27.631392 10779 leader_election.cc:302] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 22662410bfb74fa091248719a263f65f, 69a41d09bee54c1194739c2405386a2c, d09430c33e5d42eebc0403da38b85de9; no voters:
I20260620 06:02:27.631563 11004 raft_consensus.cc:2781] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:27.631820 11004 raft_consensus.cc:686] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 1 LEADER]: Becoming Leader. State: Replica: 69a41d09bee54c1194739c2405386a2c, State: Running, Role: LEADER
I20260620 06:02:27.631978 11004 consensus_queue.cc:237] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } }
I20260620 06:02:27.632560 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.632288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:40107)
0620 06:02:27.632424 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.632485 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 06:02:27.633982 10438 catalog_manager.cc:5495] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c reported cstate change: term changed from 0 to 1, leader changed from <none> to 69a41d09bee54c1194739c2405386a2c (127.3.115.3). New cstate: current_term: 1 leader_uuid: "69a41d09bee54c1194739c2405386a2c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69a41d09bee54c1194739c2405386a2c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35649 } health_report { overall_health: HEALTHY } } }
I20260620 06:02:27.649569 10995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.646380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59066 (local address 127.3.115.3:35649)
0620 06:02:27.646521 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:27.646525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:27.646540 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:27.646624 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:27.646627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:27.646683 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:27.646789 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:27.647033 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.647561 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.648452 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:27.648742 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:27.648852 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:27.649091 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:27.649172 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:27.649256 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:27.649376 (+ 120us) server_negotiation.cc:299] Negotiation successful
0620 06:02:27.649430 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:02:27.651927 10571 raft_consensus.cc:1261] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Refusing update from remote peer 69a41d09bee54c1194739c2405386a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:27.651973 10708 raft_consensus.cc:1261] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Refusing update from remote peer 69a41d09bee54c1194739c2405386a2c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:27.652470 11008 consensus_queue.cc:1035] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "22662410bfb74fa091248719a263f65f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 45025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:27.652639 11004 consensus_queue.cc:1035] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09430c33e5d42eebc0403da38b85de9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:27.654868 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.654372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:35619)
0620 06:02:27.654703 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.654779 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235}
I20260620 06:02:27.655246 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.654983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:37829)
0620 06:02:27.655100 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.655165 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 06:02:27.657882 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.657592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:33103)
0620 06:02:27.657749 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.657806 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260620 06:02:27.657884 11013 mvcc.cc:205] Tried to move back new op lower bound from 7298807183977594880 to 7298807183902281728. Current Snapshot: MvccSnapshot[applied={T|T < 7298807183977594880}]
I20260620 06:02:27.658612 11007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.658365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.3:36561)
0620 06:02:27.658497 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.658542 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 06:02:27.658915 11006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.657699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.2:59439)
0620 06:02:27.658792 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.658847 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1025}
I20260620 06:02:27.659674 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.659245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35659 (local address 127.3.115.1:39063)
0620 06:02:27.659517 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.659600 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:35659: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210}
I20260620 06:02:27.661614 11012 mvcc.cc:205] Tried to move back new op lower bound from 7298807183977594880 to 7298807183902281728. Current Snapshot: MvccSnapshot[applied={T|T < 7298807183977594880}]
I20260620 06:02:27.665448 10438 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:27.665591 10438 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55550:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:27.669615 10438 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e9f073c9072d49fba384d895994c557c: Sending DeleteTablet for 3 replicas of tablet 4e3f9ad7a976474489e0a29c9aad0e72
I20260620 06:02:27.670351 10688 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3f9ad7a976474489e0a29c9aad0e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:27 UTC) from {username='oryx', principal='oryx/127.3.115.60@KRBTEST.COM'} at 127.0.0.1:50878
I20260620 06:02:27.670461 10551 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3f9ad7a976474489e0a29c9aad0e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:27 UTC) from {username='oryx', principal='oryx/127.3.115.60@KRBTEST.COM'} at 127.0.0.1:38548
I20260620 06:02:27.670624 10826 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3f9ad7a976474489e0a29c9aad0e72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:27 UTC) from {username='oryx', principal='oryx/127.3.115.60@KRBTEST.COM'} at 127.0.0.1:59064
I20260620 06:02:27.670737 11021 tablet_replica.cc:330] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: stopping tablet replica
I20260620 06:02:27.670797 11022 tablet_replica.cc:330] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f: stopping tablet replica
I20260620 06:02:27.670877 11021 raft_consensus.cc:2229] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:27.670928 11022 raft_consensus.cc:2229] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:27.671021 11021 raft_consensus.cc:2258] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:27.671065 11022 raft_consensus.cc:2258] T 4e3f9ad7a976474489e0a29c9aad0e72 P 22662410bfb74fa091248719a263f65f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:27.671144 11023 tablet_replica.cc:330] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: stopping tablet replica
I20260620 06:02:27.671501 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 10399
I20260620 06:02:27.672469 11023 raft_consensus.cc:2229] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:27.672757 11023 raft_consensus.cc:2258] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:27.673439 11023 ts_tablet_manager.cc:1905] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:27.674461 11021 ts_tablet_manager.cc:1905] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:27.675526 11023 ts_tablet_manager.cc:1918] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:27.675618 11023 log.cc:1196] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/4e3f9ad7a976474489e0a29c9aad0e72
I20260620 06:02:27.675887 11023 ts_tablet_manager.cc:1939] T 4e3f9ad7a976474489e0a29c9aad0e72 P 69a41d09bee54c1194739c2405386a2c: Deleting consensus metadata
I20260620 06:02:27.676564 10425 catalog_manager.cc:4841] TS 69a41d09bee54c1194739c2405386a2c (127.3.115.3:35649): tablet 4e3f9ad7a976474489e0a29c9aad0e72 (table test-table [id=36a6f0b50e214242b9d15d3e435b39b5]) successfully deleted
I20260620 06:02:27.677662 11021 ts_tablet_manager.cc:1918] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:27.677750 11021 log.cc:1196] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/4e3f9ad7a976474489e0a29c9aad0e72
I20260620 06:02:27.678035 11021 ts_tablet_manager.cc:1939] T 4e3f9ad7a976474489e0a29c9aad0e72 P d09430c33e5d42eebc0403da38b85de9: Deleting consensus metadata
I20260620 06:02:27.678913 10422 catalog_manager.cc:4841] TS d09430c33e5d42eebc0403da38b85de9 (127.3.115.2:46583): tablet 4e3f9ad7a976474489e0a29c9aad0e72 (table test-table [id=36a6f0b50e214242b9d15d3e435b39b5]) successfully deleted
I20260620 06:02:27.679286 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 10625
I20260620 06:02:27.680049 10980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:27.679433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:45025 (local address 127.0.0.1:38564)
0620 06:02:27.679935 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:27.679981 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:45025: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422}
W20260620 06:02:27.680239 10424 catalog_manager.cc:4584] TS 22662410bfb74fa091248719a263f65f (127.3.115.1:45025): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e3f9ad7a976474489e0a29c9aad0e72: Network error: Client connection negotiation failed: client connection to 127.3.115.1:45025: connect: Connection refused (error 111)
I20260620 06:02:27.685504 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 10761
I20260620 06:02:27.692382 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 10116
I20260620 06:02:27.698235 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 10227
2026-06-20T06:02:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7173 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:27 dist-test-slave-4h14 krb5kdc[11026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:27 dist-test-slave-4h14 krb5kdc[11026](info): set up 2 sockets
Jun 20 06:02:27 dist-test-slave-4h14 krb5kdc[11026](info): commencing operation
krb5kdc: starting...
W20260620 06:02:28.757944 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:28 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab.
Entry for principal oryx/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab.
I20260620 06:02:28.884315 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:32815
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab
--principal=oryx/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:28.999693 11039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:29.000042 11039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:29.000120 11039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:29.004381 11039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:29.004496 11039 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 06:02:29.004526 11039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:29.004568 11039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:29.004629 11039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:29.004688 11039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:29.010313 11039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:32815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.62.keytab
--principal=oryx/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:29.011489 11039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:29.012559 11039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:29.021201 11045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.021245 11047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.021202 11044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:29.021550 11039 server_base.cc:1034] running on GCE node
I20260620 06:02:29.022156 11039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:29.023320 11039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:29.024490 11039 hybrid_clock.cc:648] HybridClock initialized: now 1781935349024458 us; error 50 us; skew 500 ppm
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:29.027446 11039 init.cc:377] Logged in from keytab as oryx/127.3.115.62@KRBTEST.COM (short username oryx)
I20260620 06:02:29.028463 11039 webserver.cc:460] Webserver started at http://127.3.115.62:33691/ using document root <none> and password file <none>
I20260620 06:02:29.029099 11039 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:29.029166 11039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:29.029393 11039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:29.031507 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "04d30683f6da45fa823542890f70d7be"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "8da36442f871f97f55201442808af3a9"
server_key_iv: "af7abf16243218d8e8f740c454e22776"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.032102 11039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "04d30683f6da45fa823542890f70d7be"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "8da36442f871f97f55201442808af3a9"
server_key_iv: "af7abf16243218d8e8f740c454e22776"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.036089 11039 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:29.038681 11054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.040102 11039 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:29.040262 11039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "04d30683f6da45fa823542890f70d7be"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "8da36442f871f97f55201442808af3a9"
server_key_iv: "af7abf16243218d8e8f740c454e22776"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.040382 11039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:29.055928 11039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:29.059063 11039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:29.059267 11039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:29.067554 11039 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:32815
I20260620 06:02:29.067564 11116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:32815 every 8 connection(s)
I20260620 06:02:29.068502 11039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:29.071100 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11039
I20260620 06:02:29.071316 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:29.071508 11117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:29.071650 3532 external_mini_cluster.cc:1428] Setting key a7894e68d25bd3557f0a3e68aaa0d983
I20260620 06:02:29.079121 11117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Bootstrap starting.
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:29.081821 11117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:29.082751 11117 log.cc:824] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:29.085307 11117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: No bootstrap required, opened a new log
I20260620 06:02:29.087970 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.073087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.3.115.62:32815)
0620 06:02:29.073513 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.073532 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.073577 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:02:29.074402 (+ 825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.074426 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.074864 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.075251 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.076108 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.077642 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.078515 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.079197 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.079363 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.082198 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.082205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.082221 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.082253 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.084840 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.085299 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.085304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.085310 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.085454 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.085740 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.085744 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.085745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.086242 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.086434 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.086703 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.087002 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":108,"threads_started":1}
I20260620 06:02:29.089349 11117 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d30683f6da45fa823542890f70d7be" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 32815 } }
I20260620 06:02:29.089566 11117 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:29.089637 11117 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d30683f6da45fa823542890f70d7be, State: Initialized, Role: FOLLOWER
I20260620 06:02:29.090058 11117 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d30683f6da45fa823542890f70d7be" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 32815 } }
I20260620 06:02:29.090241 11117 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:29.090309 11117 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:29.090399 11117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:29.091284 11117 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d30683f6da45fa823542890f70d7be" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 32815 } }
I20260620 06:02:29.091583 11117 leader_election.cc:302] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d30683f6da45fa823542890f70d7be; no voters:
I20260620 06:02:29.091861 11117 leader_election.cc:288] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:29.092002 11122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:29.092229 11122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [term 1 LEADER]: Becoming Leader. State: Replica: 04d30683f6da45fa823542890f70d7be, State: Running, Role: LEADER
I20260620 06:02:29.092540 11122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d30683f6da45fa823542890f70d7be" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 32815 } }
I20260620 06:02:29.093117 11117 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:29.094393 11124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d30683f6da45fa823542890f70d7be. Latest consensus state: current_term: 1 leader_uuid: "04d30683f6da45fa823542890f70d7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d30683f6da45fa823542890f70d7be" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 32815 } } }
I20260620 06:02:29.094421 11123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d30683f6da45fa823542890f70d7be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d30683f6da45fa823542890f70d7be" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 32815 } } }
I20260620 06:02:29.094537 11124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:29.094959 11131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:29.094542 11123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:29.098358 11131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:29.104528 11131 catalog_manager.cc:1347] Generated new cluster ID: 4fb52714c8d441ce9d850655d9f1c353
I20260620 06:02:29.104590 11131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:29.150743 11131 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:29.151786 11131 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:29.159775 11131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Generated new TSK 0
I20260620 06:02:29.160570 11131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab.
Entry for principal oryx/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab.
I20260620 06:02:29.193926 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:32815
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:29.314231 11143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:29.314491 11143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:29.314565 11143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:29.318563 11143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:29.318625 11143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:29.318663 11143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:02:29.318733 11143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:29.324200 11143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:29.325325 11143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:29.326365 11143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:29.336254 11149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.336287 11148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.336254 11151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:29.336498 11143 server_base.cc:1034] running on GCE node
I20260620 06:02:29.336978 11143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:29.337689 11143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:29.338843 11143 hybrid_clock.cc:648] HybridClock initialized: now 1781935349338805 us; error 53 us; skew 500 ppm
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:29.341794 11143 init.cc:377] Logged in from keytab as oryx/127.3.115.1@KRBTEST.COM (short username oryx)
I20260620 06:02:29.342631 11143 webserver.cc:460] Webserver started at http://127.3.115.1:38281/ using document root <none> and password file <none>
I20260620 06:02:29.343204 11143 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:29.343276 11143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:29.343513 11143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:29.345734 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "a391cd49d808461dbb2e469a04eb7c33"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "31db78ce0172c8474c2260c57917afbd"
server_key_iv: "94e4d50a46ca4dc9cc1b10ae706be6a8"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.346313 11143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "a391cd49d808461dbb2e469a04eb7c33"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "31db78ce0172c8474c2260c57917afbd"
server_key_iv: "94e4d50a46ca4dc9cc1b10ae706be6a8"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.350186 11143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:29.352980 11158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.354352 11143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:02:29.354480 11143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "a391cd49d808461dbb2e469a04eb7c33"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "31db78ce0172c8474c2260c57917afbd"
server_key_iv: "94e4d50a46ca4dc9cc1b10ae706be6a8"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.354589 11143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:29.369637 11143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:29.372565 11143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:29.372813 11143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:29.374254 11143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:29.374303 11143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.374366 11143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:29.374399 11143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:29.385216 11143 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35347
I20260620 06:02:29.385305 11273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35347 every 8 connection(s)
I20260620 06:02:29.386145 11143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:29.389156 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.375228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:36305 (local address 127.3.115.62:32815)
0620 06:02:29.375352 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.375355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.376373 (+ 1018us) server_negotiation.cc:407] Connection header received
0620 06:02:29.377677 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.377681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.377745 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.377834 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.379475 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.380348 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.382235 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.382458 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.382541 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.385780 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.385784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.385787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.385821 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.387371 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.387805 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.387808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.387809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.387885 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.388175 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.388177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.388178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.388383 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.388459 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.388931 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.389034 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:02:29.389937 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.375758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.3.115.1:36305)
0620 06:02:29.376230 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.376272 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.377422 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.378069 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.378081 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.378551 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:29.379289 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.379312 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.380491 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.380498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.382123 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.382135 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.382619 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.382623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.382836 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.383574 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:29.383600 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:29.385609 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:29.387467 (+ 1858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.387471 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.387482 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.387726 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.387959 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.387961 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.387962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.388091 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.388469 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:29.388473 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:29.388744 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.388890 (+ 146us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.389119 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":137,"threads_started":1}
I20260620 06:02:29.391223 11274 heartbeater.cc:347] Connected to a master server at 127.3.115.62:32815
I20260620 06:02:29.391237 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11143
I20260620 06:02:29.391359 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:29.391371 11274 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:29.391726 3532 external_mini_cluster.cc:1428] Setting key 1bf152e42b58e26d66084aef533d8597
I20260620 06:02:29.391894 11274 heartbeater.cc:511] Master 127.3.115.62:32815 requested a full tablet report, sending...
I20260620 06:02:29.393564 11070 ts_manager.cc:194] Registered new tserver with Master: a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347)
I20260620 06:02:29.394863 11070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:36305
WARNING: no policy specified for oryx/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:02:29.402333 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.395015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:49091 (local address 127.3.115.62:32815)
0620 06:02:29.395134 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.395139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.395154 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:29.395186 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.395188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.395226 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.395324 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.395914 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.396495 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.397403 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.397623 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.397731 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.399037 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.399041 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.399043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.399064 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.400591 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.401103 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.401106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.401108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.401190 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.401471 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.401476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.401478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.401663 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.401773 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.402031 (+ 258us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.402153 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:02:29.402379 11280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.393928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.3.115.1:49091)
0620 06:02:29.394234 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.394251 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.394341 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.395374 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.395378 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.395609 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:29.395765 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.395773 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.396622 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.396625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.397287 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.397296 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.397745 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.397748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.397841 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.398512 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:29.398525 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:29.398868 (+ 343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:29.400766 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.400769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.400771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.400992 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.401292 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.401295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.401297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.401383 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.401760 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:29.401763 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:29.401871 (+ 108us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.401978 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.402211 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab.
Entry for principal oryx/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab.
I20260620 06:02:29.420285 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:32815
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:29.533344 11283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:29.533659 11283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:29.533732 11283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:29.537842 11283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:29.537928 11283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:29.537997 11283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:02:29.538095 11283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:29.543489 11283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:29.544646 11283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:29.545866 11283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:29.555642 11289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.555639 11288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.555642 11291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:29.556116 11283 server_base.cc:1034] running on GCE node
I20260620 06:02:29.556545 11283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:29.557277 11283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:29.558460 11283 hybrid_clock.cc:648] HybridClock initialized: now 1781935349558431 us; error 40 us; skew 500 ppm
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:29.561300 11283 init.cc:377] Logged in from keytab as oryx/127.3.115.2@KRBTEST.COM (short username oryx)
I20260620 06:02:29.562227 11283 webserver.cc:460] Webserver started at http://127.3.115.2:42301/ using document root <none> and password file <none>
I20260620 06:02:29.562803 11283 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:29.562865 11283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:29.563095 11283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:29.565315 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "4fac1bd4e2a442a7a7950fecf802d309"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "e5634116f61d18b98665027b5f018d98"
server_key_iv: "0eaa1dc501c2f76a59ab1f34c4d8df2c"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.565894 11283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "4fac1bd4e2a442a7a7950fecf802d309"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "e5634116f61d18b98665027b5f018d98"
server_key_iv: "0eaa1dc501c2f76a59ab1f34c4d8df2c"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.569906 11283 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260620 06:02:29.572566 11298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.574025 11283 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:29.574178 11283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "4fac1bd4e2a442a7a7950fecf802d309"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "e5634116f61d18b98665027b5f018d98"
server_key_iv: "0eaa1dc501c2f76a59ab1f34c4d8df2c"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.574301 11283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:29.590075 11283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:29.593617 11283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:29.593820 11283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:29.595356 11283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:29.595407 11283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.595477 11283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:29.595517 11283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.605628 11283 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:36067
I20260620 06:02:29.605712 11413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:36067 every 8 connection(s)
I20260620 06:02:29.606674 11283 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:29.607322 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11283
I20260620 06:02:29.607427 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:29.607802 3532 external_mini_cluster.cc:1428] Setting key cf496b3cdc373293ac4f2851752ba7b2
I20260620 06:02:29.612143 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.596610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:39119 (local address 127.3.115.62:32815)
0620 06:02:29.596777 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.596781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.597921 (+ 1140us) server_negotiation.cc:407] Connection header received
0620 06:02:29.599696 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.599700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.599769 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.599856 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.601945 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.602902 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.604790 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.605049 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.605178 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.608350 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.608354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.608356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.608383 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.609944 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.610506 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.610510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.610512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.610610 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.610997 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.611001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.611003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.611202 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.611320 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.611839 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.611972 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 06:02:29.613077 11307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.597208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.3.115.2:39119)
0620 06:02:29.597722 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.597805 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.599405 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.600360 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.600373 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.600961 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:29.601767 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.601805 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.603041 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.603045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.604620 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.604631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.605226 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.605231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.605451 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.606124 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:29.606141 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:29.608148 (+ 2007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:29.610077 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.610083 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.610098 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.610385 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.610729 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.610733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.610735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.610885 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.611320 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:29.611328 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:29.611586 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.611782 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.612142 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":139,"threads_started":1}
I20260620 06:02:29.614497 11414 heartbeater.cc:347] Connected to a master server at 127.3.115.62:32815
I20260620 06:02:29.614655 11414 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:29.615290 11414 heartbeater.cc:511] Master 127.3.115.62:32815 requested a full tablet report, sending...
I20260620 06:02:29.616487 11071 ts_manager.cc:194] Registered new tserver with Master: 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:29.617363 11071 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:39119
WARNING: no policy specified for oryx/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.3@KRBTEST.COM" created.
I20260620 06:02:29.625285 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.616844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:50629 (local address 127.3.115.62:32815)
0620 06:02:29.616965 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.616969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.617245 (+ 276us) server_negotiation.cc:407] Connection header received
0620 06:02:29.617401 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.617404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.617452 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.617543 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.618236 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.618997 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.620093 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.620339 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.620462 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.621796 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.621800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.621803 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.621837 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.623429 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.623923 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.623929 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.623931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.624016 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.624345 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.624351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.624353 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.624575 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.624710 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.624998 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.625118 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:02:29.625298 11421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.616802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.3.115.2:50629)
0620 06:02:29.617156 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.617169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.617286 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.617559 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.617562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.617927 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:29.618071 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.618078 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.619146 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.619148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.619971 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.619980 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.620462 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.620466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.620562 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.621196 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:29.621218 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:29.621649 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:29.623561 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.623565 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.623567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.623816 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.624129 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.624134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.624136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.624238 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.624709 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:29.624712 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:29.624819 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.624940 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.625096 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab.
Entry for principal oryx/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab.
I20260620 06:02:29.636193 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:32815
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:29.749874 11423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:29.750154 11423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:29.750236 11423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:29.754511 11423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:29.754573 11423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:29.754613 11423 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 06:02:29.754683 11423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:29.760180 11423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.115.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:29.761323 11423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:29.762456 11423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:29.772226 11431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.772404 11429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:29.772261 11428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:29.772503 11423 server_base.cc:1034] running on GCE node
I20260620 06:02:29.773056 11423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:29.773682 11423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:29.774860 11423 hybrid_clock.cc:648] HybridClock initialized: now 1781935349774834 us; error 37 us; skew 500 ppm
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:29.777884 11423 init.cc:377] Logged in from keytab as oryx/127.3.115.3@KRBTEST.COM (short username oryx)
I20260620 06:02:29.779107 11423 webserver.cc:460] Webserver started at http://127.3.115.3:35365/ using document root <none> and password file <none>
I20260620 06:02:29.779701 11423 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:29.779803 11423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:29.780055 11423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:29.782181 11423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "a8266bc6b5d04510a1f5827f0ef9ade0"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "8c8722ac90336f3787e6bb6bea9250d1"
server_key_iv: "5ca531911581e24b5262909e944fe2bb"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.782763 11423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "a8266bc6b5d04510a1f5827f0ef9ade0"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "8c8722ac90336f3787e6bb6bea9250d1"
server_key_iv: "5ca531911581e24b5262909e944fe2bb"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.786617 11423 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:29.789206 11438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.790414 11423 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:29.790552 11423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "a8266bc6b5d04510a1f5827f0ef9ade0"
format_stamp: "Formatted at 2026-06-20 06:02:29 on dist-test-slave-4h14"
server_key: "8c8722ac90336f3787e6bb6bea9250d1"
server_key_iv: "5ca531911581e24b5262909e944fe2bb"
server_key_version: "encryptionkey@0"
I20260620 06:02:29.790671 11423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:29.806669 11423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:29.810173 11423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:29.810395 11423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:29.811939 11423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:29.812013 11423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:29.812081 11423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:29.812129 11423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:29.823201 11423 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:39869
I20260620 06:02:29.823287 11553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:39869 every 8 connection(s)
I20260620 06:02:29.824306 11423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:29.826557 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.812807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38143 (local address 127.3.115.62:32815)
0620 06:02:29.812978 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.812985 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.813955 (+ 970us) server_negotiation.cc:407] Connection header received
0620 06:02:29.815343 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.815349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.815411 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.815512 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.817272 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.818153 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.819832 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.820071 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.820187 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.822918 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.822922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.822925 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.822957 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.824560 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.825189 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.825193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.825194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.825294 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.825632 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.825634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.825635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.825812 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.825884 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.826339 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.826440 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:02:29.827397 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.813262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.3.115.3:38143)
0620 06:02:29.813769 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.813833 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.815049 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.815746 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.815760 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.816263 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:29.817069 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.817092 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.818279 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.818283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.819709 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.819725 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.820237 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.820242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.820426 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.821127 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:29.821150 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:29.822725 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:29.824694 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.824702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.824719 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.825064 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.825388 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.825392 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.825394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.825541 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.825897 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:29.825902 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:29.826128 (+ 226us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.826277 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.826521 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":119,"threads_started":1}
I20260620 06:02:29.828563 11554 heartbeater.cc:347] Connected to a master server at 127.3.115.62:32815
I20260620 06:02:29.828730 11554 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:29.829281 11554 heartbeater.cc:511] Master 127.3.115.62:32815 requested a full tablet report, sending...
I20260620 06:02:29.830442 11070 ts_manager.cc:194] Registered new tserver with Master: a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869)
I20260620 06:02:29.831238 11070 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:38143
I20260620 06:02:29.833048 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11423
I20260620 06:02:29.833179 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:29.833469 3532 external_mini_cluster.cc:1428] Setting key a6ad0886ba19451dadcc9141c0b87afb
I20260620 06:02:29.835541 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:02:29.838429 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.830988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:45695 (local address 127.3.115.62:32815)
0620 06:02:29.831104 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.831107 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.831327 (+ 220us) server_negotiation.cc:407] Connection header received
0620 06:02:29.831471 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.831474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.831519 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.831619 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.832063 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.832568 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.833518 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.833809 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.833926 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.835203 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.835207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.835210 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.835233 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.836779 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.837207 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.837210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.837214 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.837302 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.837586 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.837591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.837593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.837778 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.837887 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.838155 (+ 268us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.838263 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:02:29.838451 11560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.830913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.3.115.3:45695)
0620 06:02:29.831239 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.831252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.831377 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.831667 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.831669 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.831848 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:29.831945 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.831951 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.832725 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.832728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.833404 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.833412 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.833950 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.833955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.834054 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.834676 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:29.834690 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:29.835072 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:29.836893 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.836897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.836900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.837104 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.837397 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:29.837402 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:29.837404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:29.837491 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:29.837875 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:29.837878 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:29.837984 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.838091 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.838259 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":92,"threads_started":1}
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:02:29.861420 11120 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:29.854712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60326 (local address 127.3.115.62:32815)
0620 06:02:29.854862 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.854866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.854879 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:02:29.854921 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.854924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.854965 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.855037 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.855787 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.856318 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.857241 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.857452 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.857564 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.861245 (+ 3681us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60326: BlockingRecv error: recv got EOF from 127.0.0.1:60326 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 06:02:29 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:29.871037 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.862795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60330 (local address 127.3.115.62:32815)
0620 06:02:29.862927 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.862931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.863064 (+ 133us) server_negotiation.cc:407] Connection header received
0620 06:02:29.863134 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.863137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.863183 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.863263 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:29.863902 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.864430 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.865512 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.865733 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.865817 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.868202 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:29.868206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:29.868209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:29.868236 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:29.869718 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.870073 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.870075 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.870077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.870148 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:29.870373 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:29.870375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:29.870376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:29.870534 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:29.870636 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.870834 (+ 198us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.870921 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:02:29.874004 11070 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:29.877074 11070 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:29.891999 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.885964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.3.115.1:35347)
0620 06:02:29.887250 (+ 1286us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.887268 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.887293 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:02:29.887372 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.887376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.887601 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.887761 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:29.888076 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.889298 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.890647 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.891293 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.891449 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.891619 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.891755 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.891851 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1161,"thread_start_us":109,"threads_started":1}
I20260620 06:02:29.891996 11579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.885824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35347 (local address 127.0.0.1:49262)
0620 06:02:29.886753 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.886810 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.886944 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.887760 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.887763 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.887779 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:29.887933 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.887941 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.889418 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.889421 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.890536 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.890543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.891457 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.891461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.891560 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.891600 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.891711 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.891833 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":25,"spinlock_wait_cycles":6272,"thread_start_us":88,"threads_started":1}
I20260620 06:02:29.891996 11577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.886139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39869 (local address 127.0.0.1:56942)
0620 06:02:29.887045 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.887059 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.887135 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.887731 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.887737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.887760 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:29.887935 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.887942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.889077 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.889083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.890446 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.890456 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.891667 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.891673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.891767 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.891786 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.891848 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.891900 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":3,"spinlock_wait_cycles":1920,"thread_start_us":88,"threads_started":1}
I20260620 06:02:29.892305 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.886621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.3.115.3:39869)
0620 06:02:29.887269 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.887279 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.887319 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:02:29.887394 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.887400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.887579 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.887721 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:29.888051 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.888975 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.890571 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.891322 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.891446 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.891608 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.892063 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.892161 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":119,"threads_started":1}
I20260620 06:02:29.892719 11576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.885832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:36067 (local address 127.0.0.1:32858)
0620 06:02:29.886753 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:29.886811 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:29.886944 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:29.888466 (+ 1522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:29.888469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:29.888487 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:29.888578 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.888583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.889871 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.889874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.891017 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:29.891027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.892289 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.892293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:29.892387 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.892409 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:02:29.892481 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:02:29.892551 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":2,"spinlock_wait_cycles":9856,"thread_start_us":91,"threads_started":1}
I20260620 06:02:29.893025 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:29.886607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32858 (local address 127.3.115.2:36067)
0620 06:02:29.888032 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:29.888045 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:29.888065 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:02:29.888133 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:29.888140 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:29.888320 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:29.888469 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:29.888712 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.889745 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.891150 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:29.891949 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:29.892056 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:29.892232 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:29.892539 (+ 307us) server_negotiation.cc:299] Negotiation successful
0620 06:02:29.892643 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1307,"thread_start_us":148,"threads_started":1}
I20260620 06:02:29.895350 11488 tablet_service.cc:1467] Processing CreateTablet for tablet 32a4b2e1c93543ca86f3ea06591512ee (DEFAULT_TABLE table=test-table [id=ab78edcdec514b468281c9f421d8db6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:29.895336 11331 tablet_service.cc:1467] Processing CreateTablet for tablet 32a4b2e1c93543ca86f3ea06591512ee (DEFAULT_TABLE table=test-table [id=ab78edcdec514b468281c9f421d8db6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:29.895334 11208 tablet_service.cc:1467] Processing CreateTablet for tablet 32a4b2e1c93543ca86f3ea06591512ee (DEFAULT_TABLE table=test-table [id=ab78edcdec514b468281c9f421d8db6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:29.896353 11488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a4b2e1c93543ca86f3ea06591512ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:29.896354 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a4b2e1c93543ca86f3ea06591512ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:29.896359 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32a4b2e1c93543ca86f3ea06591512ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:29.903317 11582 tablet_bootstrap.cc:492] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Bootstrap starting.
I20260620 06:02:29.904596 11583 tablet_bootstrap.cc:492] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Bootstrap starting.
I20260620 06:02:29.905071 11584 tablet_bootstrap.cc:492] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Bootstrap starting.
I20260620 06:02:29.905803 11582 tablet_bootstrap.cc:654] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:29.906697 11583 tablet_bootstrap.cc:654] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:29.906730 11582 log.cc:824] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:29.907114 11584 tablet_bootstrap.cc:654] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:29.907856 11584 log.cc:824] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:29.907863 11583 log.cc:824] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:29.908959 11582 tablet_bootstrap.cc:492] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: No bootstrap required, opened a new log
I20260620 06:02:29.909162 11582 ts_tablet_manager.cc:1397] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:02:29.909759 11584 tablet_bootstrap.cc:492] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: No bootstrap required, opened a new log
I20260620 06:02:29.909759 11583 tablet_bootstrap.cc:492] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: No bootstrap required, opened a new log
I20260620 06:02:29.909934 11584 ts_tablet_manager.cc:1397] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260620 06:02:29.909934 11583 ts_tablet_manager.cc:1397] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:02:29.912349 11582 raft_consensus.cc:348] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:29.912595 11582 raft_consensus.cc:374] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:29.912667 11582 raft_consensus.cc:729] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Initialized, Role: FOLLOWER
I20260620 06:02:29.913164 11582 consensus_queue.cc:260] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:29.913846 11554 heartbeater.cc:503] Master 127.3.115.62:32815 was elected leader, sending a full tablet report...
I20260620 06:02:29.913859 11583 raft_consensus.cc:348] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:29.913805 11584 raft_consensus.cc:348] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:29.914057 11584 raft_consensus.cc:374] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:29.914057 11583 raft_consensus.cc:374] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:29.914104 11583 raft_consensus.cc:729] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391cd49d808461dbb2e469a04eb7c33, State: Initialized, Role: FOLLOWER
I20260620 06:02:29.914104 11584 raft_consensus.cc:729] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fac1bd4e2a442a7a7950fecf802d309, State: Initialized, Role: FOLLOWER
I20260620 06:02:29.914250 11582 ts_tablet_manager.cc:1428] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 06:02:29.914561 11584 consensus_queue.cc:260] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:29.914594 11583 consensus_queue.cc:260] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:29.915465 11583 ts_tablet_manager.cc:1428] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:02:29.915465 11584 ts_tablet_manager.cc:1428] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260620 06:02:29.915541 11274 heartbeater.cc:503] Master 127.3.115.62:32815 was elected leader, sending a full tablet report...
I20260620 06:02:29.916850 11414 heartbeater.cc:503] Master 127.3.115.62:32815 was elected leader, sending a full tablet report...
I20260620 06:02:30.002709 11588 raft_consensus.cc:482] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:30.002974 11588 raft_consensus.cc:504] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.004220 11588 leader_election.cc:288] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:30.008431 11447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.004515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35347 (local address 127.3.115.3:36901)
0620 06:02:30.004720 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.004736 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.004842 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.005134 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.005140 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.005158 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:30.005300 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.005306 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.006129 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.006133 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.007272 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.007283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.008076 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.008079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.008162 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.008183 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.008255 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.008314 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":30}
I20260620 06:02:30.008510 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.004568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:36901 (local address 127.3.115.1:35347)
0620 06:02:30.004723 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.004730 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.004811 (+ 81us) server_negotiation.cc:407] Connection header received
0620 06:02:30.004953 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.004961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.005009 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.005103 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:30.005433 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.006002 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.007410 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.007929 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.008039 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.008105 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.008294 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.008369 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:02:30.009032 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.004719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:43819 (local address 127.3.115.2:36067)
0620 06:02:30.004846 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.004851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.005085 (+ 234us) server_negotiation.cc:407] Connection header received
0620 06:02:30.005248 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.005256 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.005298 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.005357 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:30.006001 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.006539 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.007783 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.008410 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.008513 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.008582 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.008854 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.008926 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:02:30.009045 11591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.004590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:36067 (local address 127.3.115.3:43819)
0620 06:02:30.004982 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.004993 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.005114 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.005397 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.005401 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.005421 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:30.005877 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.005883 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.006664 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.006668 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.007667 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.007674 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.008565 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.008569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.008700 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.008735 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.008824 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.008892 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"mutex_wait_us":1,"thread_start_us":197,"threads_started":1}
I20260620 06:02:30.009231 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a4b2e1c93543ca86f3ea06591512ee" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33" is_pre_election: true
I20260620 06:02:30.009615 11228 raft_consensus.cc:2454] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 0.
I20260620 06:02:30.009785 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a4b2e1c93543ca86f3ea06591512ee" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309" is_pre_election: true
I20260620 06:02:30.010089 11368 raft_consensus.cc:2454] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 0.
I20260620 06:02:30.010233 11442 leader_election.cc:302] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a391cd49d808461dbb2e469a04eb7c33, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.010552 11588 raft_consensus.cc:2781] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:30.010643 11588 raft_consensus.cc:482] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:30.010706 11588 raft_consensus.cc:3037] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.011751 11588 raft_consensus.cc:504] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.012126 11588 leader_election.cc:288] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 election: Requested vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:30.012462 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a4b2e1c93543ca86f3ea06591512ee" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309"
I20260620 06:02:30.012531 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32a4b2e1c93543ca86f3ea06591512ee" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33"
I20260620 06:02:30.012593 11368 raft_consensus.cc:3037] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.012714 11228 raft_consensus.cc:3037] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.013710 11368 raft_consensus.cc:2454] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 1.
I20260620 06:02:30.013882 11228 raft_consensus.cc:2454] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 1.
I20260620 06:02:30.014038 11441 leader_election.cc:302] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fac1bd4e2a442a7a7950fecf802d309, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.014262 11588 raft_consensus.cc:2781] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:30.014509 11588 raft_consensus.cc:686] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 LEADER]: Becoming Leader. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Running, Role: LEADER
I20260620 06:02:30.014853 11588 consensus_queue.cc:237] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.018164 11069 catalog_manager.cc:5495] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } health_report { overall_health: HEALTHY } } }
I20260620 06:02:30.055372 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.051981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.3.115.3:39869)
0620 06:02:30.052136 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.052140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.052216 (+ 76us) server_negotiation.cc:407] Connection header received
0620 06:02:30.052292 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.052294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.052350 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.052434 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:30.052665 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.053250 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.054071 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.054316 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.054405 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.054621 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.055055 (+ 434us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.055141 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.055217 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.055265 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 06:02:30.061412 11368 raft_consensus.cc:1261] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Refusing update from remote peer a8266bc6b5d04510a1f5827f0ef9ade0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.061435 11228 raft_consensus.cc:1261] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Refusing update from remote peer a8266bc6b5d04510a1f5827f0ef9ade0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.062119 11588 consensus_queue.cc:1035] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.062350 11592 consensus_queue.cc:1035] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.071331 11597 mvcc.cc:205] Tried to move back new op lower bound from 7298807193844699136 to 7298807193663733760. Current Snapshot: MvccSnapshot[applied={T|T < 7298807193844699136}]
I20260620 06:02:30.072062 11596 mvcc.cc:205] Tried to move back new op lower bound from 7298807193844699136 to 7298807193663733760. Current Snapshot: MvccSnapshot[applied={T|T < 7298807193844699136}]
I20260620 06:02:30.078169 11595 mvcc.cc:205] Tried to move back new op lower bound from 7298807193844699136 to 7298807193663733760. Current Snapshot: MvccSnapshot[applied={T|T < 7298807193844699136}]
I20260620 06:02:30.086917 11068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.087100 11068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.089572 11068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Sending DeleteTablet for 3 replicas of tablet 32a4b2e1c93543ca86f3ea06591512ee
I20260620 06:02:30.090363 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a4b2e1c93543ca86f3ea06591512ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 06:02:30.090381 11208 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a4b2e1c93543ca86f3ea06591512ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49262
I20260620 06:02:30.090457 11488 tablet_service.cc:1514] Processing DeleteTablet for tablet 32a4b2e1c93543ca86f3ea06591512ee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260620 06:02:30.090914 11605 tablet_replica.cc:330] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: stopping tablet replica
I20260620 06:02:30.091130 11607 tablet_replica.cc:330] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: stopping tablet replica
I20260620 06:02:30.091234 11605 raft_consensus.cc:2229] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.091523 11607 raft_consensus.cc:2229] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.091562 11605 raft_consensus.cc:2258] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.091594 11068 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60330:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:30.092029 11068 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:30.092150 11607 raft_consensus.cc:2258] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.092906 11606 tablet_replica.cc:330] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: stopping tablet replica
I20260620 06:02:30.093150 11606 raft_consensus.cc:2229] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:30.093366 11605 ts_tablet_manager.cc:1905] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.093525 11606 raft_consensus.cc:2258] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.094695 11607 ts_tablet_manager.cc:1905] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.096293 11605 ts_tablet_manager.cc:1918] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:30.096390 11605 log.cc:1196] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/32a4b2e1c93543ca86f3ea06591512ee
I20260620 06:02:30.096742 11605 ts_tablet_manager.cc:1939] T 32a4b2e1c93543ca86f3ea06591512ee P 4fac1bd4e2a442a7a7950fecf802d309: Deleting consensus metadata
I20260620 06:02:30.097481 11607 ts_tablet_manager.cc:1918] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:30.097579 11607 log.cc:1196] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/32a4b2e1c93543ca86f3ea06591512ee
I20260620 06:02:30.097865 11607 ts_tablet_manager.cc:1939] T 32a4b2e1c93543ca86f3ea06591512ee P a391cd49d808461dbb2e469a04eb7c33: Deleting consensus metadata
I20260620 06:02:30.098264 11331 tablet_service.cc:1467] Processing CreateTablet for tablet d57b21beebc549d0bf45b48119952018 (DEFAULT_TABLE table=test-table [id=7a7e125e464e44928c031dd53f402984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.098336 11208 tablet_service.cc:1467] Processing CreateTablet for tablet d57b21beebc549d0bf45b48119952018 (DEFAULT_TABLE table=test-table [id=7a7e125e464e44928c031dd53f402984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.098500 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57b21beebc549d0bf45b48119952018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.098567 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57b21beebc549d0bf45b48119952018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.099116 11057 catalog_manager.cc:4841] TS 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067): tablet 32a4b2e1c93543ca86f3ea06591512ee (table test-table [id=ab78edcdec514b468281c9f421d8db6d]) successfully deleted
I20260620 06:02:30.099350 11606 ts_tablet_manager.cc:1905] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.100591 11583 tablet_bootstrap.cc:492] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Bootstrap starting.
I20260620 06:02:30.100730 11584 tablet_bootstrap.cc:492] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Bootstrap starting.
I20260620 06:02:30.100903 11058 catalog_manager.cc:4841] TS a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347): tablet 32a4b2e1c93543ca86f3ea06591512ee (table test-table [id=ab78edcdec514b468281c9f421d8db6d]) successfully deleted
I20260620 06:02:30.101701 11584 tablet_bootstrap.cc:654] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.101936 11583 tablet_bootstrap.cc:654] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.101998 11606 ts_tablet_manager.cc:1918] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:30.102087 11606 log.cc:1196] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/32a4b2e1c93543ca86f3ea06591512ee
I20260620 06:02:30.102402 11606 ts_tablet_manager.cc:1939] T 32a4b2e1c93543ca86f3ea06591512ee P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting consensus metadata
I20260620 06:02:30.102888 11488 tablet_service.cc:1467] Processing CreateTablet for tablet d57b21beebc549d0bf45b48119952018 (DEFAULT_TABLE table=test-table [id=7a7e125e464e44928c031dd53f402984]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.103118 11488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57b21beebc549d0bf45b48119952018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.103528 11056 catalog_manager.cc:4841] TS a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869): tablet 32a4b2e1c93543ca86f3ea06591512ee (table test-table [id=ab78edcdec514b468281c9f421d8db6d]) successfully deleted
I20260620 06:02:30.103839 11583 tablet_bootstrap.cc:492] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: No bootstrap required, opened a new log
I20260620 06:02:30.103907 11583 ts_tablet_manager.cc:1397] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:30.104358 11584 tablet_bootstrap.cc:492] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: No bootstrap required, opened a new log
I20260620 06:02:30.104432 11584 ts_tablet_manager.cc:1397] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:02:30.104712 11583 raft_consensus.cc:348] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.104820 11583 raft_consensus.cc:374] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.104848 11583 raft_consensus.cc:729] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391cd49d808461dbb2e469a04eb7c33, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.104864 11584 raft_consensus.cc:348] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.104977 11584 raft_consensus.cc:374] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.104959 11583 consensus_queue.cc:260] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.105013 11584 raft_consensus.cc:729] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fac1bd4e2a442a7a7950fecf802d309, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.105140 11584 consensus_queue.cc:260] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.105202 11583 ts_tablet_manager.cc:1428] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.105356 11582 tablet_bootstrap.cc:492] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Bootstrap starting.
I20260620 06:02:30.105414 11584 ts_tablet_manager.cc:1428] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.106274 11582 tablet_bootstrap.cc:654] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.107643 11582 tablet_bootstrap.cc:492] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: No bootstrap required, opened a new log
I20260620 06:02:30.107705 11582 ts_tablet_manager.cc:1397] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 06:02:30.108199 11582 raft_consensus.cc:348] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.108356 11582 raft_consensus.cc:374] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.108397 11582 raft_consensus.cc:729] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.108549 11582 consensus_queue.cc:260] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
W20260620 06:02:30.108862 11415 tablet.cc:2369] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:30.109417 11582 ts_tablet_manager.cc:1428] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260620 06:02:30.137830 11275 tablet.cc:2369] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:30.147851 11601 raft_consensus.cc:482] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:30.148012 11601 raft_consensus.cc:504] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.148451 11601 leader_election.cc:288] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:30.148897 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57b21beebc549d0bf45b48119952018" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33" is_pre_election: true
I20260620 06:02:30.148897 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57b21beebc549d0bf45b48119952018" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309" is_pre_election: true
I20260620 06:02:30.149055 11228 raft_consensus.cc:2454] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 0.
I20260620 06:02:30.149055 11368 raft_consensus.cc:2454] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 0.
I20260620 06:02:30.149333 11442 leader_election.cc:302] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a391cd49d808461dbb2e469a04eb7c33, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.149493 11601 raft_consensus.cc:2781] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:30.149554 11601 raft_consensus.cc:482] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:30.149592 11601 raft_consensus.cc:3037] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.150966 11601 raft_consensus.cc:504] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.151358 11601 leader_election.cc:288] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 election: Requested vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:30.151666 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57b21beebc549d0bf45b48119952018" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33"
I20260620 06:02:30.151736 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57b21beebc549d0bf45b48119952018" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309"
I20260620 06:02:30.151818 11228 raft_consensus.cc:3037] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.151863 11368 raft_consensus.cc:3037] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.152947 11228 raft_consensus.cc:2454] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 1.
I20260620 06:02:30.153024 11368 raft_consensus.cc:2454] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 1.
I20260620 06:02:30.153303 11442 leader_election.cc:302] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a391cd49d808461dbb2e469a04eb7c33, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.153486 11601 raft_consensus.cc:2781] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:30.153578 11601 raft_consensus.cc:686] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 LEADER]: Becoming Leader. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Running, Role: LEADER
I20260620 06:02:30.153743 11601 consensus_queue.cc:237] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.155612 11071 catalog_manager.cc:5495] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } health_report { overall_health: HEALTHY } } }
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935349, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.62@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:30.200371 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.190394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:32815 (local address 127.0.0.1:60342)
0620 06:02:30.190685 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.190711 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.190839 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.191189 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.191192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.191717 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:30.191881 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.191892 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.192869 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.192875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.193789 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.193801 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.194403 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.194408 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.194512 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.195226 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:30.195240 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:30.196958 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:30.198764 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:30.198775 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:30.198781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:30.199015 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:30.199292 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:30.199297 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:30.199299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:30.199382 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:30.199773 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:30.199781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:30.199895 (+ 114us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.200009 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.200170 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":110,"threads_started":1}
I20260620 06:02:30.200371 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.190470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60342 (local address 127.3.115.62:32815)
0620 06:02:30.190613 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.190617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.190796 (+ 179us) server_negotiation.cc:407] Connection header received
0620 06:02:30.190950 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.190952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.191038 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.191125 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:30.192028 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.192704 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.193921 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.194276 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.194367 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.197100 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:30.197103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:30.197106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:30.197127 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:30.198618 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:30.199122 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:30.199126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:30.199127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:30.199203 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:30.199475 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:30.199478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:30.199480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:30.199652 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:30.199773 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.200065 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.200225 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":65}
I20260620 06:02:30.204459 11071 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:60342:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 06:02:30.205237 11071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:30.210492 11208 tablet_service.cc:1467] Processing CreateTablet for tablet ac7d5e93bb1a40e0a178528d877b162d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23cdc385cf094726890b0eda91118628]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:02:30.210553 11331 tablet_service.cc:1467] Processing CreateTablet for tablet ac7d5e93bb1a40e0a178528d877b162d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23cdc385cf094726890b0eda91118628]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:02:30.210553 11488 tablet_service.cc:1467] Processing CreateTablet for tablet ac7d5e93bb1a40e0a178528d877b162d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=23cdc385cf094726890b0eda91118628]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 06:02:30.210955 11488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7d5e93bb1a40e0a178528d877b162d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.210955 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7d5e93bb1a40e0a178528d877b162d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.210960 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac7d5e93bb1a40e0a178528d877b162d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.213903 11584 tablet_bootstrap.cc:492] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: Bootstrap starting.
I20260620 06:02:30.213902 11582 tablet_bootstrap.cc:492] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: Bootstrap starting.
I20260620 06:02:30.214216 11583 tablet_bootstrap.cc:492] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: Bootstrap starting.
I20260620 06:02:30.214990 11584 tablet_bootstrap.cc:654] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.215227 11583 tablet_bootstrap.cc:654] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.215348 11582 tablet_bootstrap.cc:654] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.216732 11583 tablet_bootstrap.cc:492] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: No bootstrap required, opened a new log
I20260620 06:02:30.216818 11583 ts_tablet_manager.cc:1397] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:30.217203 11582 tablet_bootstrap.cc:492] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: No bootstrap required, opened a new log
I20260620 06:02:30.217278 11582 ts_tablet_manager.cc:1397] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:30.217322 11583 raft_consensus.cc:348] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.217410 11583 raft_consensus.cc:374] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.217430 11583 raft_consensus.cc:729] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391cd49d808461dbb2e469a04eb7c33, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.217571 11583 consensus_queue.cc:260] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.217705 11582 raft_consensus.cc:348] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.217816 11582 raft_consensus.cc:374] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.217851 11582 raft_consensus.cc:729] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.217852 11583 ts_tablet_manager.cc:1428] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.217809 11589 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.217931 11589 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.217968 11582 consensus_queue.cc:260] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.218214 11582 ts_tablet_manager.cc:1428] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.218326 11584 tablet_bootstrap.cc:492] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: No bootstrap required, opened a new log
I20260620 06:02:30.218405 11584 ts_tablet_manager.cc:1397] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260620 06:02:30.218420 11601 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.218533 11601 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.218818 11584 raft_consensus.cc:348] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.218924 11584 raft_consensus.cc:374] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.218956 11584 raft_consensus.cc:729] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fac1bd4e2a442a7a7950fecf802d309, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.219089 11584 consensus_queue.cc:260] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.219363 11584 ts_tablet_manager.cc:1428] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.219550 11590 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.219671 11590 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.318210 11589 raft_consensus.cc:482] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:30.318470 11589 raft_consensus.cc:504] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.319558 11589 leader_election.cc:288] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067), a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869)
I20260620 06:02:30.323648 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.320013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:48081 (local address 127.3.115.2:36067)
0620 06:02:30.320138 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.320143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.320429 (+ 286us) server_negotiation.cc:407] Connection header received
0620 06:02:30.320539 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.320542 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.320611 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.320742 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:30.321013 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.321497 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.322683 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.323073 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.323182 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.323252 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.323432 (+ 180us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.323512 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:02:30.323658 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.319942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37903 (local address 127.3.115.3:39869)
0620 06:02:30.320084 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.320088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.320345 (+ 257us) server_negotiation.cc:407] Connection header received
0620 06:02:30.320442 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.320445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.320502 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.320580 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:30.320949 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.321472 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.322715 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.323093 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.323214 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.323279 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.323484 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.323558 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:02:30.323679 11622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.319818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:36067 (local address 127.3.115.1:48081)
0620 06:02:30.320285 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.320320 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.320443 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.320746 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.320748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.320761 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:30.320899 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.320907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.321622 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.321625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.322558 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.322567 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.323193 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.323205 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.323302 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.323331 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.323390 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.323475 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":203,"thread_start_us":86,"threads_started":1}
I20260620 06:02:30.323678 11621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.319818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39869 (local address 127.3.115.1:37903)
0620 06:02:30.320210 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.320245 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.320346 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.320603 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.320606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.320622 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:30.320824 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.320834 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.321606 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.321609 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.322562 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.322571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.323244 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.323248 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.323343 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.323359 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.323455 (+ 96us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.323514 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":40,"thread_start_us":122,"threads_started":1}
I20260620 06:02:30.324231 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7d5e93bb1a40e0a178528d877b162d" candidate_uuid: "a391cd49d808461dbb2e469a04eb7c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309" is_pre_election: true
I20260620 06:02:30.324407 11368 raft_consensus.cc:2454] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a391cd49d808461dbb2e469a04eb7c33 in term 0.
I20260620 06:02:30.324528 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7d5e93bb1a40e0a178528d877b162d" candidate_uuid: "a391cd49d808461dbb2e469a04eb7c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" is_pre_election: true
I20260620 06:02:30.324821 11161 leader_election.cc:302] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fac1bd4e2a442a7a7950fecf802d309, a391cd49d808461dbb2e469a04eb7c33; no voters:
I20260620 06:02:30.324872 11508 raft_consensus.cc:2454] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a391cd49d808461dbb2e469a04eb7c33 in term 0.
I20260620 06:02:30.325102 11589 raft_consensus.cc:2781] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:30.325168 11589 raft_consensus.cc:482] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:30.325208 11589 raft_consensus.cc:3037] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Advancing to term 1
W20260620 06:02:30.325949 11555 tablet.cc:2369] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:02:30.326264 11589 raft_consensus.cc:504] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.326653 11589 leader_election.cc:288] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [CANDIDATE]: Term 1 election: Requested vote from peers 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067), a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869)
I20260620 06:02:30.327000 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7d5e93bb1a40e0a178528d877b162d" candidate_uuid: "a391cd49d808461dbb2e469a04eb7c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309"
I20260620 06:02:30.327035 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac7d5e93bb1a40e0a178528d877b162d" candidate_uuid: "a391cd49d808461dbb2e469a04eb7c33" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0"
I20260620 06:02:30.327104 11368 raft_consensus.cc:3037] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.327165 11508 raft_consensus.cc:3037] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.327991 11368 raft_consensus.cc:2454] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a391cd49d808461dbb2e469a04eb7c33 in term 1.
I20260620 06:02:30.328135 11508 raft_consensus.cc:2454] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a391cd49d808461dbb2e469a04eb7c33 in term 1.
I20260620 06:02:30.328282 11161 leader_election.cc:302] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fac1bd4e2a442a7a7950fecf802d309, a391cd49d808461dbb2e469a04eb7c33; no voters:
I20260620 06:02:30.328452 11589 raft_consensus.cc:2781] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:30.328727 11589 raft_consensus.cc:686] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [term 1 LEADER]: Becoming Leader. State: Replica: a391cd49d808461dbb2e469a04eb7c33, State: Running, Role: LEADER
I20260620 06:02:30.329015 11589 consensus_queue.cc:237] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.330178 11623 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: TxnStatusTablet state changed. Reason: New leader a391cd49d808461dbb2e469a04eb7c33. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.330295 11623 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:02:30.330554 11625 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 06:02:30.330703 11625 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 06:02:30.331496 11069 catalog_manager.cc:5495] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 reported cstate change: term changed from 0 to 1, leader changed from <none> to a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } health_report { overall_health: UNKNOWN } } }
I20260620 06:02:30.350798 11617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.346555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35347 (local address 127.0.0.1:49274)
0620 06:02:30.346745 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.346763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.346882 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.347191 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.347194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.347202 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:30.347294 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.347298 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.348301 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.348304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.348989 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.348998 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.349563 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.349565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.349630 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.349672 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.350499 (+ 827us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.350515 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.350588 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.350679 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 06:02:30.350811 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.346736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.3.115.1:35347)
0620 06:02:30.346882 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.346887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.346903 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:02:30.347000 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.347003 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.347066 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.347169 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:30.347395 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.348168 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.349098 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.349451 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.349550 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.349819 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.350376 (+ 557us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.350493 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.350619 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.350680 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:02:30.353881 11368 raft_consensus.cc:1261] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Refusing update from remote peer a391cd49d808461dbb2e469a04eb7c33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.354058 11508 raft_consensus.cc:1261] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Refusing update from remote peer a391cd49d808461dbb2e469a04eb7c33: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.354414 11623 consensus_queue.cc:1035] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.354820 11589 consensus_queue.cc:1035] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.356516 11601 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: TxnStatusTablet state changed. Reason: New leader a391cd49d808461dbb2e469a04eb7c33. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.356621 11601 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.357147 11590 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: TxnStatusTablet state changed. Reason: New leader a391cd49d808461dbb2e469a04eb7c33. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.357275 11590 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.358927 11589 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.359031 11589 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:02:30.359851 11623 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.359947 11623 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a391cd49d808461dbb2e469a04eb7c33: This TxnStatusTablet replica's current role is: LEADER
I20260620 06:02:30.360123 11601 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.360237 11601 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P a8266bc6b5d04510a1f5827f0ef9ade0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.360996 11590 tablet_replica.cc:439] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a391cd49d808461dbb2e469a04eb7c33" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } } }
I20260620 06:02:30.361101 11590 tablet_replica.cc:442] T ac7d5e93bb1a40e0a178528d877b162d P 4fac1bd4e2a442a7a7950fecf802d309: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 06:02:30.374631 11632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.371091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35347 (local address 127.3.115.3:45059)
0620 06:02:30.371373 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.371386 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.371475 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.371714 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.371716 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.371722 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:30.371809 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.371813 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.372833 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.372836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.373446 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.373453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.374001 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.374003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.374065 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.374107 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.374397 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.374401 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.374452 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.374518 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":109,"threads_started":1}
I20260620 06:02:30.374641 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.371186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:45059 (local address 127.3.115.1:35347)
0620 06:02:30.371316 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.371323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.371455 (+ 132us) server_negotiation.cc:407] Connection header received
0620 06:02:30.371547 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.371550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.371607 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.371687 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:30.371903 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.372736 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.373558 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.373895 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.373996 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.374238 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.374317 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.374388 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.374463 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.374512 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:02:30.381217 11368 raft_consensus.cc:1261] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Refusing update from remote peer a8266bc6b5d04510a1f5827f0ef9ade0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.381353 11228 raft_consensus.cc:1261] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Refusing update from remote peer a8266bc6b5d04510a1f5827f0ef9ade0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.381646 11601 consensus_queue.cc:1035] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.381820 11588 consensus_queue.cc:1035] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.407342 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.402835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:47929 (local address 127.3.115.3:39869)
0620 06:02:30.402958 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.402963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.403241 (+ 278us) server_negotiation.cc:407] Connection header received
0620 06:02:30.403442 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.403445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.403505 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.403558 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:30.403968 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.404802 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.406055 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.406442 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.406550 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.406887 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.406970 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.407042 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.407138 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.407191 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:02:30.407362 11637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.402745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39869 (local address 127.3.115.1:47929)
0620 06:02:30.403158 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.403174 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.403318 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.403714 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.403717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.403725 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 06:02:30.403845 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.403852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.404932 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.404935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.405950 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.405960 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.406570 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.406574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.406669 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.406751 (+ 82us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.407062 (+ 311us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.407068 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.407129 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.407211 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":110,"threads_started":1}
I20260620 06:02:30.438138 11069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.438318 11069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60330:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.440601 11069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Sending DeleteTablet for 3 replicas of tablet d57b21beebc549d0bf45b48119952018
I20260620 06:02:30.441179 11208 tablet_service.cc:1514] Processing DeleteTablet for tablet d57b21beebc549d0bf45b48119952018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49262
I20260620 06:02:30.441268 11488 tablet_service.cc:1514] Processing DeleteTablet for tablet d57b21beebc549d0bf45b48119952018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260620 06:02:30.441318 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet d57b21beebc549d0bf45b48119952018 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 06:02:30.441371 11607 tablet_replica.cc:330] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: stopping tablet replica
I20260620 06:02:30.441442 11606 tablet_replica.cc:330] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: stopping tablet replica
I20260620 06:02:30.441471 11605 tablet_replica.cc:330] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: stopping tablet replica
I20260620 06:02:30.441474 11607 raft_consensus.cc:2229] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.441545 11606 raft_consensus.cc:2229] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:30.441569 11605 raft_consensus.cc:2229] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.441586 11607 raft_consensus.cc:2258] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.441673 11605 raft_consensus.cc:2258] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.441748 11606 raft_consensus.cc:2258] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.442458 11605 ts_tablet_manager.cc:1905] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.442649 11607 ts_tablet_manager.cc:1905] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.445080 11605 ts_tablet_manager.cc:1918] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:30.445173 11605 log.cc:1196] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/d57b21beebc549d0bf45b48119952018
I20260620 06:02:30.445366 11607 ts_tablet_manager.cc:1918] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:30.445443 11605 ts_tablet_manager.cc:1939] T d57b21beebc549d0bf45b48119952018 P 4fac1bd4e2a442a7a7950fecf802d309: Deleting consensus metadata
I20260620 06:02:30.445447 11607 log.cc:1196] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/d57b21beebc549d0bf45b48119952018
I20260620 06:02:30.445700 11607 ts_tablet_manager.cc:1939] T d57b21beebc549d0bf45b48119952018 P a391cd49d808461dbb2e469a04eb7c33: Deleting consensus metadata
I20260620 06:02:30.446190 11057 catalog_manager.cc:4841] TS 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067): tablet d57b21beebc549d0bf45b48119952018 (table test-table [id=7a7e125e464e44928c031dd53f402984]) successfully deleted
I20260620 06:02:30.446372 11058 catalog_manager.cc:4841] TS a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347): tablet d57b21beebc549d0bf45b48119952018 (table test-table [id=7a7e125e464e44928c031dd53f402984]) successfully deleted
I20260620 06:02:30.446601 11606 ts_tablet_manager.cc:1905] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.448742 11606 ts_tablet_manager.cc:1918] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:30.448829 11606 log.cc:1196] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/d57b21beebc549d0bf45b48119952018
I20260620 06:02:30.449093 11606 ts_tablet_manager.cc:1939] T d57b21beebc549d0bf45b48119952018 P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting consensus metadata
I20260620 06:02:30.449836 11056 catalog_manager.cc:4841] TS a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869): tablet d57b21beebc549d0bf45b48119952018 (table test-table [id=7a7e125e464e44928c031dd53f402984]) successfully deleted
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:02:30.475186 11120 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:30.467989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.3.115.62:32815)
0620 06:02:30.468149 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.468154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.468335 (+ 181us) server_negotiation.cc:407] Connection header received
0620 06:02:30.468436 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.468439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.468495 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.468589 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:30.469396 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.470244 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.471262 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.471580 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.471702 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.475053 (+ 3351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60344: BlockingRecv error: recv got EOF from 127.0.0.1:60344 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11026](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM
I20260620 06:02:30.486052 11120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.476905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60356 (local address 127.3.115.62:32815)
0620 06:02:30.477025 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.477034 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.477157 (+ 123us) server_negotiation.cc:407] Connection header received
0620 06:02:30.477296 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.477298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.477338 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.477441 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:30.478193 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.478820 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.479951 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.480161 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.480246 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.482859 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:30.482862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:30.482865 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:30.482888 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:30.484345 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:30.484838 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:30.484842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:30.484844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:30.484925 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:30.485204 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:30.485208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:30.485209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:30.485388 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:30.485528 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.485768 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.485887 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:02:30.488869 11069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:30.489348 11069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:30.494158 11208 tablet_service.cc:1467] Processing CreateTablet for tablet cc57ac0131dc4ff88b07e94776e8616d (DEFAULT_TABLE table=test-table [id=8457697fbf2441ffbfd2bb22a7bffb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.494494 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc57ac0131dc4ff88b07e94776e8616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.494455 11331 tablet_service.cc:1467] Processing CreateTablet for tablet cc57ac0131dc4ff88b07e94776e8616d (DEFAULT_TABLE table=test-table [id=8457697fbf2441ffbfd2bb22a7bffb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.494706 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc57ac0131dc4ff88b07e94776e8616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.496954 11488 tablet_service.cc:1467] Processing CreateTablet for tablet cc57ac0131dc4ff88b07e94776e8616d (DEFAULT_TABLE table=test-table [id=8457697fbf2441ffbfd2bb22a7bffb60]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.497017 11584 tablet_bootstrap.cc:492] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Bootstrap starting.
I20260620 06:02:30.497223 11488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc57ac0131dc4ff88b07e94776e8616d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.497961 11584 tablet_bootstrap.cc:654] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.499154 11582 tablet_bootstrap.cc:492] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Bootstrap starting.
I20260620 06:02:30.499181 11584 tablet_bootstrap.cc:492] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: No bootstrap required, opened a new log
I20260620 06:02:30.499289 11584 ts_tablet_manager.cc:1397] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:30.499716 11584 raft_consensus.cc:348] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.499830 11584 raft_consensus.cc:374] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.499859 11584 raft_consensus.cc:729] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fac1bd4e2a442a7a7950fecf802d309, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.499991 11582 tablet_bootstrap.cc:654] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.499974 11584 consensus_queue.cc:260] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.500200 11584 ts_tablet_manager.cc:1428] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.500748 11583 tablet_bootstrap.cc:492] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Bootstrap starting.
I20260620 06:02:30.501597 11583 tablet_bootstrap.cc:654] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.502465 11582 tablet_bootstrap.cc:492] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: No bootstrap required, opened a new log
I20260620 06:02:30.502568 11582 ts_tablet_manager.cc:1397] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:02:30.502902 11582 raft_consensus.cc:348] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.503015 11582 raft_consensus.cc:374] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.503053 11582 raft_consensus.cc:729] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.503194 11582 consensus_queue.cc:260] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.503479 11582 ts_tablet_manager.cc:1428] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:02:30.504007 11583 tablet_bootstrap.cc:492] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: No bootstrap required, opened a new log
I20260620 06:02:30.504122 11583 ts_tablet_manager.cc:1397] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:02:30.504467 11583 raft_consensus.cc:348] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.504580 11583 raft_consensus.cc:374] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.504619 11583 raft_consensus.cc:729] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391cd49d808461dbb2e469a04eb7c33, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.504768 11583 consensus_queue.cc:260] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.505024 11583 ts_tablet_manager.cc:1428] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 06:02:30.569706 11590 raft_consensus.cc:482] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:30.569839 11590 raft_consensus.cc:504] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.571137 11590 leader_election.cc:288] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869)
I20260620 06:02:30.576479 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.571599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:50631 (local address 127.3.115.1:35347)
0620 06:02:30.571728 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.571732 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.571919 (+ 187us) server_negotiation.cc:407] Connection header received
0620 06:02:30.572077 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.572080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.572131 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.572220 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:30.572711 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.573511 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.575220 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.575847 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.575961 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.576045 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.576282 (+ 237us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.576352 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:02:30.576503 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.571523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:50597 (local address 127.3.115.3:39869)
0620 06:02:30.571675 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.571679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.572020 (+ 341us) server_negotiation.cc:407] Connection header received
0620 06:02:30.572078 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.572081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.572131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.572224 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:30.572711 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.573494 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.575360 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.575970 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.576061 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.576125 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.576257 (+ 132us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.576334 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 06:02:30.576558 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.571376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35347 (local address 127.3.115.2:50631)
0620 06:02:30.571823 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.571839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.571957 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.572282 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.572286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.572310 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:30.572570 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.572578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.573650 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.573657 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.575091 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.575102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.575984 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.575990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.576113 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.576143 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.576226 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.576303 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":54,"spinlock_wait_cycles":7424,"thread_start_us":99,"threads_started":1}
I20260620 06:02:30.577010 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.571378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39869 (local address 127.3.115.2:50597)
0620 06:02:30.571908 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:30.571921 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:30.572009 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:30.572282 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:30.572309 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:30.572322 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:30.572570 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.572578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.574096 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.574099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.575215 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:30.575222 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.576059 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.576062 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:30.576142 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.576158 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 06:02:30.576226 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 06:02:30.576866 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":75,"spinlock_wait_cycles":768,"thread_start_us":91,"threads_started":1}
I20260620 06:02:30.577131 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc57ac0131dc4ff88b07e94776e8616d" candidate_uuid: "4fac1bd4e2a442a7a7950fecf802d309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33" is_pre_election: true
I20260620 06:02:30.577287 11228 raft_consensus.cc:2454] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fac1bd4e2a442a7a7950fecf802d309 in term 0.
I20260620 06:02:30.577486 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc57ac0131dc4ff88b07e94776e8616d" candidate_uuid: "4fac1bd4e2a442a7a7950fecf802d309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" is_pre_election: true
I20260620 06:02:30.577654 11508 raft_consensus.cc:2454] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fac1bd4e2a442a7a7950fecf802d309 in term 0.
I20260620 06:02:30.577667 11302 leader_election.cc:302] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fac1bd4e2a442a7a7950fecf802d309, a391cd49d808461dbb2e469a04eb7c33; no voters:
I20260620 06:02:30.577962 11590 raft_consensus.cc:2781] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:30.578047 11590 raft_consensus.cc:482] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:30.578088 11590 raft_consensus.cc:3037] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.579550 11590 raft_consensus.cc:504] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.579970 11590 leader_election.cc:288] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [CANDIDATE]: Term 1 election: Requested vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869)
I20260620 06:02:30.580284 11508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc57ac0131dc4ff88b07e94776e8616d" candidate_uuid: "4fac1bd4e2a442a7a7950fecf802d309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0"
I20260620 06:02:30.580291 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc57ac0131dc4ff88b07e94776e8616d" candidate_uuid: "4fac1bd4e2a442a7a7950fecf802d309" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33"
I20260620 06:02:30.580397 11508 raft_consensus.cc:3037] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.580436 11228 raft_consensus.cc:3037] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.581576 11508 raft_consensus.cc:2454] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fac1bd4e2a442a7a7950fecf802d309 in term 1.
I20260620 06:02:30.581738 11228 raft_consensus.cc:2454] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fac1bd4e2a442a7a7950fecf802d309 in term 1.
I20260620 06:02:30.581856 11300 leader_election.cc:302] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fac1bd4e2a442a7a7950fecf802d309, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.582037 11590 raft_consensus.cc:2781] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:30.582252 11590 raft_consensus.cc:686] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 LEADER]: Becoming Leader. State: Replica: 4fac1bd4e2a442a7a7950fecf802d309, State: Running, Role: LEADER
I20260620 06:02:30.582523 11590 consensus_queue.cc:237] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.584877 11068 catalog_manager.cc:5495] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2). New cstate: current_term: 1 leader_uuid: "4fac1bd4e2a442a7a7950fecf802d309" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } health_report { overall_health: UNKNOWN } } }
I20260620 06:02:30.608151 11581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.604258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32862 (local address 127.3.115.2:36067)
0620 06:02:30.604388 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.604392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.604405 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:02:30.604484 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.604485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.604537 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.604631 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:30.604974 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.605800 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.606680 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.606977 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.607081 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.607315 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.607819 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.607925 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.607977 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.608026 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:02:30.610803 11508 raft_consensus.cc:1261] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Refusing update from remote peer 4fac1bd4e2a442a7a7950fecf802d309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.610811 11228 raft_consensus.cc:1261] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Refusing update from remote peer 4fac1bd4e2a442a7a7950fecf802d309: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.611306 11656 consensus_queue.cc:1035] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.611480 11590 consensus_queue.cc:1035] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.623435 11068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.623591 11068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.625723 11068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Sending DeleteTablet for 3 replicas of tablet cc57ac0131dc4ff88b07e94776e8616d
I20260620 06:02:30.626214 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet cc57ac0131dc4ff88b07e94776e8616d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 06:02:30.626216 11208 tablet_service.cc:1514] Processing DeleteTablet for tablet cc57ac0131dc4ff88b07e94776e8616d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49262
I20260620 06:02:30.626374 11605 tablet_replica.cc:330] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: stopping tablet replica
I20260620 06:02:30.626390 11607 tablet_replica.cc:330] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: stopping tablet replica
I20260620 06:02:30.626474 11605 raft_consensus.cc:2229] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:30.626494 11607 raft_consensus.cc:2229] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.626616 11607 raft_consensus.cc:2258] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.626724 11605 raft_consensus.cc:2258] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.627357 11605 ts_tablet_manager.cc:1905] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.627378 11607 ts_tablet_manager.cc:1905] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.629832 11607 ts_tablet_manager.cc:1918] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:30.629848 11605 ts_tablet_manager.cc:1918] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:30.629904 11607 log.cc:1196] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/cc57ac0131dc4ff88b07e94776e8616d
I20260620 06:02:30.629917 11605 log.cc:1196] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/cc57ac0131dc4ff88b07e94776e8616d
I20260620 06:02:30.630172 11607 ts_tablet_manager.cc:1939] T cc57ac0131dc4ff88b07e94776e8616d P a391cd49d808461dbb2e469a04eb7c33: Deleting consensus metadata
I20260620 06:02:30.630162 11605 ts_tablet_manager.cc:1939] T cc57ac0131dc4ff88b07e94776e8616d P 4fac1bd4e2a442a7a7950fecf802d309: Deleting consensus metadata
I20260620 06:02:30.630776 11057 catalog_manager.cc:4841] TS 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067): tablet cc57ac0131dc4ff88b07e94776e8616d (table test-table [id=8457697fbf2441ffbfd2bb22a7bffb60]) successfully deleted
I20260620 06:02:30.630795 11058 catalog_manager.cc:4841] TS a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347): tablet cc57ac0131dc4ff88b07e94776e8616d (table test-table [id=8457697fbf2441ffbfd2bb22a7bffb60]) successfully deleted
I20260620 06:02:30.631402 11488 tablet_service.cc:1514] Processing DeleteTablet for tablet cc57ac0131dc4ff88b07e94776e8616d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260620 06:02:30.631577 11606 tablet_replica.cc:330] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: stopping tablet replica
I20260620 06:02:30.631681 11606 raft_consensus.cc:2229] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.631805 11606 raft_consensus.cc:2258] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.632508 11606 ts_tablet_manager.cc:1905] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.632838 11071 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:30.633226 11071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:30.634898 11606 ts_tablet_manager.cc:1918] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:30.634984 11606 log.cc:1196] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/cc57ac0131dc4ff88b07e94776e8616d
I20260620 06:02:30.635232 11606 ts_tablet_manager.cc:1939] T cc57ac0131dc4ff88b07e94776e8616d P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting consensus metadata
I20260620 06:02:30.635998 11056 catalog_manager.cc:4841] TS a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869): tablet cc57ac0131dc4ff88b07e94776e8616d (table test-table [id=8457697fbf2441ffbfd2bb22a7bffb60]) successfully deleted
I20260620 06:02:30.637995 11208 tablet_service.cc:1467] Processing CreateTablet for tablet cdb27ffaafe9428d818b82843e5cf9f2 (DEFAULT_TABLE table=test-table [id=155688099b6049adaa2f474a5cd7fa20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.638074 11331 tablet_service.cc:1467] Processing CreateTablet for tablet cdb27ffaafe9428d818b82843e5cf9f2 (DEFAULT_TABLE table=test-table [id=155688099b6049adaa2f474a5cd7fa20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.638340 11208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb27ffaafe9428d818b82843e5cf9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.638367 11331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb27ffaafe9428d818b82843e5cf9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.640225 11584 tablet_bootstrap.cc:492] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Bootstrap starting.
I20260620 06:02:30.640590 11583 tablet_bootstrap.cc:492] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Bootstrap starting.
I20260620 06:02:30.641198 11584 tablet_bootstrap.cc:654] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.641283 11488 tablet_service.cc:1467] Processing CreateTablet for tablet cdb27ffaafe9428d818b82843e5cf9f2 (DEFAULT_TABLE table=test-table [id=155688099b6049adaa2f474a5cd7fa20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:30.641426 11583 tablet_bootstrap.cc:654] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.641517 11488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdb27ffaafe9428d818b82843e5cf9f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:30.642871 11583 tablet_bootstrap.cc:492] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: No bootstrap required, opened a new log
I20260620 06:02:30.642979 11583 ts_tablet_manager.cc:1397] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 06:02:30.643347 11583 raft_consensus.cc:348] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.643465 11583 raft_consensus.cc:374] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.643502 11583 raft_consensus.cc:729] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a391cd49d808461dbb2e469a04eb7c33, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.643641 11583 consensus_queue.cc:260] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.643946 11583 ts_tablet_manager.cc:1428] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.644243 11582 tablet_bootstrap.cc:492] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Bootstrap starting.
I20260620 06:02:30.644397 11584 tablet_bootstrap.cc:492] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: No bootstrap required, opened a new log
I20260620 06:02:30.644477 11584 ts_tablet_manager.cc:1397] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 06:02:30.644977 11584 raft_consensus.cc:348] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.645107 11584 raft_consensus.cc:374] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.645152 11582 tablet_bootstrap.cc:654] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:30.645150 11584 raft_consensus.cc:729] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fac1bd4e2a442a7a7950fecf802d309, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.645290 11584 consensus_queue.cc:260] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.645587 11584 ts_tablet_manager.cc:1428] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.646900 11582 tablet_bootstrap.cc:492] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: No bootstrap required, opened a new log
I20260620 06:02:30.647001 11582 ts_tablet_manager.cc:1397] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:30.647444 11582 raft_consensus.cc:348] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.647564 11582 raft_consensus.cc:374] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:30.647605 11582 raft_consensus.cc:729] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Initialized, Role: FOLLOWER
I20260620 06:02:30.647748 11582 consensus_queue.cc:260] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.648046 11582 ts_tablet_manager.cc:1428] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 06:02:30.748167 11601 raft_consensus.cc:482] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:30.748329 11601 raft_consensus.cc:504] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.748872 11601 leader_election.cc:288] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:30.749366 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb27ffaafe9428d818b82843e5cf9f2" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309" is_pre_election: true
I20260620 06:02:30.749392 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb27ffaafe9428d818b82843e5cf9f2" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33" is_pre_election: true
I20260620 06:02:30.749531 11368 raft_consensus.cc:2454] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 0.
I20260620 06:02:30.749536 11228 raft_consensus.cc:2454] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 0.
I20260620 06:02:30.749825 11442 leader_election.cc:302] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a391cd49d808461dbb2e469a04eb7c33, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.750063 11601 raft_consensus.cc:2781] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:30.750134 11601 raft_consensus.cc:482] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:30.750177 11601 raft_consensus.cc:3037] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.751499 11601 raft_consensus.cc:504] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.751894 11601 leader_election.cc:288] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 election: Requested vote from peers a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347), 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067)
I20260620 06:02:30.752285 11368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb27ffaafe9428d818b82843e5cf9f2" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fac1bd4e2a442a7a7950fecf802d309"
I20260620 06:02:30.752281 11228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdb27ffaafe9428d818b82843e5cf9f2" candidate_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a391cd49d808461dbb2e469a04eb7c33"
I20260620 06:02:30.752426 11228 raft_consensus.cc:3037] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.752431 11368 raft_consensus.cc:3037] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:30.753578 11368 raft_consensus.cc:2454] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 1.
I20260620 06:02:30.753595 11228 raft_consensus.cc:2454] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8266bc6b5d04510a1f5827f0ef9ade0 in term 1.
I20260620 06:02:30.753877 11442 leader_election.cc:302] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a391cd49d808461dbb2e469a04eb7c33, a8266bc6b5d04510a1f5827f0ef9ade0; no voters:
I20260620 06:02:30.754057 11601 raft_consensus.cc:2781] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:30.754155 11601 raft_consensus.cc:686] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 LEADER]: Becoming Leader. State: Replica: a8266bc6b5d04510a1f5827f0ef9ade0, State: Running, Role: LEADER
I20260620 06:02:30.754295 11601 consensus_queue.cc:237] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } }
I20260620 06:02:30.756002 11071 catalog_manager.cc:5495] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8266bc6b5d04510a1f5827f0ef9ade0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39869 } health_report { overall_health: HEALTHY } } }
I20260620 06:02:30.777987 11580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:30.774335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.3.115.3:39869)
0620 06:02:30.774445 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:30.774450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:30.774529 (+ 79us) server_negotiation.cc:407] Connection header received
0620 06:02:30.774637 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:30.774640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:30.774687 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:30.774771 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:30.775005 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.775819 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.776712 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:30.777062 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:30.777160 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:30.777494 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:30.777573 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:30.777638 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:30.777777 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 06:02:30.777827 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:02:30.784035 11368 raft_consensus.cc:1261] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Refusing update from remote peer a8266bc6b5d04510a1f5827f0ef9ade0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.784086 11228 raft_consensus.cc:1261] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Refusing update from remote peer a8266bc6b5d04510a1f5827f0ef9ade0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:30.784443 11601 consensus_queue.cc:1035] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fac1bd4e2a442a7a7950fecf802d309" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 36067 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.784686 11592 consensus_queue.cc:1035] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a391cd49d808461dbb2e469a04eb7c33" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:30.812419 11069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.812568 11069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:30.814402 11069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d30683f6da45fa823542890f70d7be: Sending DeleteTablet for 3 replicas of tablet cdb27ffaafe9428d818b82843e5cf9f2
I20260620 06:02:30.814889 11208 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb27ffaafe9428d818b82843e5cf9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49262
I20260620 06:02:30.814990 11488 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb27ffaafe9428d818b82843e5cf9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260620 06:02:30.815070 11607 tablet_replica.cc:330] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: stopping tablet replica
I20260620 06:02:30.815140 11606 tablet_replica.cc:330] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: stopping tablet replica
I20260620 06:02:30.815176 11607 raft_consensus.cc:2229] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.815229 11606 raft_consensus.cc:2229] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:30.815310 11607 raft_consensus.cc:2258] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.815392 11606 raft_consensus.cc:2258] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.815897 11331 tablet_service.cc:1514] Processing DeleteTablet for tablet cdb27ffaafe9428d818b82843e5cf9f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:30 UTC) from {username='oryx', principal='oryx/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:32858
I20260620 06:02:30.816067 11605 tablet_replica.cc:330] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: stopping tablet replica
I20260620 06:02:30.816124 11607 ts_tablet_manager.cc:1905] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.816129 11606 ts_tablet_manager.cc:1905] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.816179 11605 raft_consensus.cc:2229] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:30.816282 11605 raft_consensus.cc:2258] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:30.816900 11605 ts_tablet_manager.cc:1905] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:30.818189 11606 ts_tablet_manager.cc:1918] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:30.818246 11606 log.cc:1196] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/cdb27ffaafe9428d818b82843e5cf9f2
I20260620 06:02:30.818427 11606 ts_tablet_manager.cc:1939] T cdb27ffaafe9428d818b82843e5cf9f2 P a8266bc6b5d04510a1f5827f0ef9ade0: Deleting consensus metadata
I20260620 06:02:30.818694 11607 ts_tablet_manager.cc:1918] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:30.818760 11607 log.cc:1196] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/cdb27ffaafe9428d818b82843e5cf9f2
I20260620 06:02:30.818991 11607 ts_tablet_manager.cc:1939] T cdb27ffaafe9428d818b82843e5cf9f2 P a391cd49d808461dbb2e469a04eb7c33: Deleting consensus metadata
I20260620 06:02:30.819048 11056 catalog_manager.cc:4841] TS a8266bc6b5d04510a1f5827f0ef9ade0 (127.3.115.3:39869): tablet cdb27ffaafe9428d818b82843e5cf9f2 (table test-table [id=155688099b6049adaa2f474a5cd7fa20]) successfully deleted
I20260620 06:02:30.819525 11605 ts_tablet_manager.cc:1918] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 06:02:30.819625 11605 log.cc:1196] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/cdb27ffaafe9428d818b82843e5cf9f2
I20260620 06:02:30.819660 11058 catalog_manager.cc:4841] TS a391cd49d808461dbb2e469a04eb7c33 (127.3.115.1:35347): tablet cdb27ffaafe9428d818b82843e5cf9f2 (table test-table [id=155688099b6049adaa2f474a5cd7fa20]) successfully deleted
I20260620 06:02:30.819891 11605 ts_tablet_manager.cc:1939] T cdb27ffaafe9428d818b82843e5cf9f2 P 4fac1bd4e2a442a7a7950fecf802d309: Deleting consensus metadata
I20260620 06:02:30.820495 11057 catalog_manager.cc:4841] TS 4fac1bd4e2a442a7a7950fecf802d309 (127.3.115.2:36067): tablet cdb27ffaafe9428d818b82843e5cf9f2 (table test-table [id=155688099b6049adaa2f474a5cd7fa20]) successfully deleted
I20260620 06:02:30.820623 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11143
I20260620 06:02:30.830302 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11283
I20260620 06:02:30.839977 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11423
I20260620 06:02:30.847877 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11039
2026-06-20T06:02:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3148 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11672](info): set up 2 sockets
Jun 20 06:02:30 dist-test-slave-4h14 krb5kdc[11672](info): commencing operation
krb5kdc: starting...
W20260620 06:02:31.893752 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:31 dist-test-slave-4h14 krb5kdc[11672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:32.013923 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40945
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:46103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:40945
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:32.123541 11685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:32.123884 11685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:32.123960 11685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:32.128080 11685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:32.128194 11685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:32.128232 11685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:32.128278 11685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:32.128319 11685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:32.133685 11685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:46103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:40945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:32.134806 11685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:32.135846 11685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:32.144306 11690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:32.144310 11693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:32.144313 11691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:32.144690 11685 server_base.cc:1034] running on GCE node
I20260620 06:02:32.145390 11685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:32.146498 11685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:32.147722 11685 hybrid_clock.cc:648] HybridClock initialized: now 1781935352147708 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T06:02:32Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1302 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:32 dist-test-slave-4h14 krb5kdc[11698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:32 dist-test-slave-4h14 krb5kdc[11698](info): set up 2 sockets
Jun 20 06:02:32 dist-test-slave-4h14 krb5kdc[11698](info): commencing operation
krb5kdc: starting...
W20260620 06:02:33.197724 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:33 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:33.321676 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40765
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:40765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:33.433231 11711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:33.433528 11711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:33.433570 11711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:33.437628 11711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:33.437690 11711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:33.437711 11711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:33.437727 11711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:33.437742 11711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:33.442977 11711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:40765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:33.444005 11711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:33.445062 11711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:33.453662 11716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:33.453670 11719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:33.453881 11711 server_base.cc:1034] running on GCE node
W20260620 06:02:33.453792 11717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:33.454579 11711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:33.455730 11711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:33.456902 11711 hybrid_clock.cc:648] HybridClock initialized: now 1781935353456882 us; error 35 us; skew 500 ppm
Jun 20 06:02:33 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:33.464195 11711 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:33.465313 11711 webserver.cc:460] Webserver started at http://127.3.115.62:36627/ using document root <none> and password file <none>
I20260620 06:02:33.465900 11711 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:33.465992 11711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:33.466230 11711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:33.468323 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "0333b9f4b165746070923e76ae3388c4"
server_key_iv: "b1a519255ee7e72c85d3fa640acd8a31"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.468904 11711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "0333b9f4b165746070923e76ae3388c4"
server_key_iv: "b1a519255ee7e72c85d3fa640acd8a31"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.472700 11711 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:02:33.475231 11726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.476583 11711 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:33.476763 11711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "0333b9f4b165746070923e76ae3388c4"
server_key_iv: "b1a519255ee7e72c85d3fa640acd8a31"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.476882 11711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:33.486191 11711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:33.489464 11711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:33.489665 11711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:33.497084 11711 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:40765
I20260620 06:02:33.497102 11778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:40765 every 8 connection(s)
I20260620 06:02:33.498013 11711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:33.500705 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:33.506084 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Bootstrap starting.
I20260620 06:02:33.508066 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11711
I20260620 06:02:33.508200 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:33.508472 3532 external_mini_cluster.cc:1428] Setting key 291993de9b4f5e4a5ab8145c8419a2ee
I20260620 06:02:33.508628 11779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:33.509390 11779 log.cc:824] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:33.511346 11779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: No bootstrap required, opened a new log
I20260620 06:02:33.515599 11779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:33.515895 11779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:33.515955 11779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b2d8fb0b4d48f88c554e5d9f1b8e89, State: Initialized, Role: FOLLOWER
I20260620 06:02:33.516508 11779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:33.516700 11779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:33.516781 11779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:33.516902 11779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 06:02:33 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:33.518126 11779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:33.518471 11779 leader_election.cc:302] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b2d8fb0b4d48f88c554e5d9f1b8e89; no voters:
I20260620 06:02:33.518810 11779 leader_election.cc:288] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:33.518932 11784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:33.519151 11784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 LEADER]: Becoming Leader. State: Replica: 23b2d8fb0b4d48f88c554e5d9f1b8e89, State: Running, Role: LEADER
I20260620 06:02:33.519749 11784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:33.520174 11779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:33.521627 11785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b2d8fb0b4d48f88c554e5d9f1b8e89. Latest consensus state: current_term: 1 leader_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } } }
I20260620 06:02:33.521721 11785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:33.522084 11786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } } }
I20260620 06:02:33.522188 11786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:33.522756 11794 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:33.523974 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:33.509670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.3.115.62:40765)
0620 06:02:33.510056 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:33.510072 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:33.510105 (+ 33us) server_negotiation.cc:407] Connection header received
0620 06:02:33.510730 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:33.510746 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:33.511061 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:33.511346 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:33.512246 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:33.513307 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:33.514545 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:33.515045 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:33.515168 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:33.518160 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:33.518166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:33.518180 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:33.518241 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:33.520354 (+ 2113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:33.520901 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:33.520908 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:33.520914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:33.521068 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:33.521405 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:33.521409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:33.521411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:33.521953 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:33.522169 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:33.522682 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 06:02:33.522994 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":110,"threads_started":1}
I20260620 06:02:33.525804 11794 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:33.531145 11794 catalog_manager.cc:1347] Generated new cluster ID: 7d26e17e26474e6c8662e279422ec54b
I20260620 06:02:33.531205 11794 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:33.545817 11794 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:33.546823 11794 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:33.555192 11794 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Generated new TSK 0
I20260620 06:02:33.556051 11794 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:33.585810 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:40765
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:33.698951 11805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:33.699241 11805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:33.699330 11805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:33.703439 11805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:33.703507 11805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:33.703608 11805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:33.709086 11805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:33.710227 11805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:33.711380 11805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:33.720922 11813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:33.721020 11805 server_base.cc:1034] running on GCE node
W20260620 06:02:33.720906 11810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:33.720906 11811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:33.721660 11805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:33.722329 11805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:33.723519 11805 hybrid_clock.cc:648] HybridClock initialized: now 1781935353723485 us; error 46 us; skew 500 ppm
Jun 20 06:02:33 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:33.726363 11805 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:33.727191 11805 webserver.cc:460] Webserver started at http://127.3.115.1:40897/ using document root <none> and password file <none>
I20260620 06:02:33.727695 11805 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:33.727751 11805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:33.727919 11805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:33.729929 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "2a607b2307b24f38b7cfeb018a514f65"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "7bc6ccdcd74fefa5cedc689174e94b50"
server_key_iv: "3e5b9f1147d6f084d1de384c3c022cd6"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.730409 11805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "2a607b2307b24f38b7cfeb018a514f65"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "7bc6ccdcd74fefa5cedc689174e94b50"
server_key_iv: "3e5b9f1147d6f084d1de384c3c022cd6"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.734043 11805 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:02:33.736407 11820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.737649 11805 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:33.737741 11805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "2a607b2307b24f38b7cfeb018a514f65"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "7bc6ccdcd74fefa5cedc689174e94b50"
server_key_iv: "3e5b9f1147d6f084d1de384c3c022cd6"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.737885 11805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:33.773020 11805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:33.776548 11805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:33.776793 11805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:33.777431 11805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:33.778328 11805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:33.778376 11805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.778442 11805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:33.778472 11805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.787465 11805 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:42019
I20260620 06:02:33.787530 11933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:42019 every 8 connection(s)
I20260620 06:02:33.788416 11805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:33.792827 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11805
I20260620 06:02:33.792972 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:33.793247 3532 external_mini_cluster.cc:1428] Setting key 51ece6f6fd65c58fe4f642bb5ec3617a
Jun 20 06:02:33 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:02:33.803354 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:33.790278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:48639 (local address 127.3.115.62:40765)
0620 06:02:33.790424 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:33.790427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:33.791320 (+ 893us) server_negotiation.cc:407] Connection header received
0620 06:02:33.792607 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:33.792611 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:33.792705 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:33.792813 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:33.794269 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:33.794998 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:33.796188 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:33.796420 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:33.796529 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:33.799408 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:33.799415 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:33.799421 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:33.799456 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:33.801237 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:33.801738 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:33.801742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:33.801746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:33.801837 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:33.802165 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:33.802171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:33.802173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:33.802419 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:33.802536 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:33.803007 (+ 471us) server_negotiation.cc:299] Negotiation successful
0620 06:02:33.803129 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:02:33.804106 11936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:33.790652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:48639)
0620 06:02:33.791146 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:33.791212 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:33.792393 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:33.793055 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:33.793064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:33.793480 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:33.794118 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:33.794137 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:33.795136 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:33.795139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:33.796060 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:33.796069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:33.796568 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:33.796572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:33.796790 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:33.797407 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:33.797423 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:33.799234 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:33.801358 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:33.801362 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:33.801375 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:33.801630 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:33.801934 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:33.801936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:33.801938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:33.802068 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:33.802527 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:33.802532 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:33.802761 (+ 229us) client_negotiation.cc:769] Sending connection context
0620 06:02:33.802926 (+ 165us) client_negotiation.cc:240] Negotiation successful
0620 06:02:33.803176 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":98,"threads_started":1}
I20260620 06:02:33.805325 11934 heartbeater.cc:347] Connected to a master server at 127.3.115.62:40765
I20260620 06:02:33.805568 11934 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:33.806062 11934 heartbeater.cc:511] Master 127.3.115.62:40765 requested a full tablet report, sending...
I20260620 06:02:33.807550 11743 ts_manager.cc:194] Registered new tserver with Master: 2a607b2307b24f38b7cfeb018a514f65 (127.3.115.1:42019)
I20260620 06:02:33.808703 11743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:48639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:33.820191 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:40765
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:33.933558 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:33.933835 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:33.933880 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:33.937873 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:33.937932 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:33.937991 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:33.943372 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:33.944396 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:33.945374 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:33.954974 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:33.954969 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:33.954969 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:33.955170 11939 server_base.cc:1034] running on GCE node
I20260620 06:02:33.955555 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:33.956195 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:33.957376 11939 hybrid_clock.cc:648] HybridClock initialized: now 1781935353957354 us; error 40 us; skew 500 ppm
Jun 20 06:02:33 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:33.960127 11939 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:33.961037 11939 webserver.cc:460] Webserver started at http://127.3.115.2:32777/ using document root <none> and password file <none>
I20260620 06:02:33.961553 11939 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:33.961611 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:33.961768 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:33.963841 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "c687cfe3f2344082ba4a38165ef48a1a"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "00f622ec51f952c3675259e4e8bdc148"
server_key_iv: "be146b8a47441acd2c74dfce99cdcdab"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.964357 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "c687cfe3f2344082ba4a38165ef48a1a"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "00f622ec51f952c3675259e4e8bdc148"
server_key_iv: "be146b8a47441acd2c74dfce99cdcdab"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.968008 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:02:33.970588 11954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.971903 11939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:02:33.972004 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "c687cfe3f2344082ba4a38165ef48a1a"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "00f622ec51f952c3675259e4e8bdc148"
server_key_iv: "be146b8a47441acd2c74dfce99cdcdab"
server_key_version: "encryptionkey@0"
I20260620 06:02:33.972097 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:33.983511 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:33.986874 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:33.987032 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:33.987550 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:33.988309 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:33.988358 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.988396 11939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:33.988411 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:33.997795 11939 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:35103
I20260620 06:02:33.997817 12067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:35103 every 8 connection(s)
I20260620 06:02:33.998740 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:34.006536 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 11939
I20260620 06:02:34.006680 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:34.006950 3532 external_mini_cluster.cc:1428] Setting key 2adc08c67bd378e94d7873cec297eb62
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935353, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:34.013468 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.000620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:49885 (local address 127.3.115.62:40765)
0620 06:02:34.000805 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.000809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.001669 (+ 860us) server_negotiation.cc:407] Connection header received
0620 06:02:34.003407 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.003411 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.003516 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.003619 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.004860 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.005387 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.006560 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.006888 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.007000 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.009754 (+ 2754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:34.009761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:34.009766 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:34.009797 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:34.011351 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:34.011879 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:34.011883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:34.011886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:34.011968 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:34.012312 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:34.012315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:34.012317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:34.012563 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:34.012688 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:34.013180 (+ 492us) server_negotiation.cc:299] Negotiation successful
0620 06:02:34.013306 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 06:02:34.014365 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.001058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:49885)
0620 06:02:34.001506 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.001549 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.003107 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.003819 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.003827 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.004200 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.004699 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.004713 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.005517 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.005521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.006428 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.006437 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.007039 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.007043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.007224 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.007926 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.007946 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.009594 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:34.011457 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:34.011462 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:34.011475 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:34.011755 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:34.012060 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:34.012065 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:34.012067 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:34.012206 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:34.012688 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:34.012696 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:34.012940 (+ 244us) client_negotiation.cc:769] Sending connection context
0620 06:02:34.013124 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 06:02:34.013458 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260620 06:02:34.015655 12068 heartbeater.cc:347] Connected to a master server at 127.3.115.62:40765
I20260620 06:02:34.015897 12068 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:34.016369 12068 heartbeater.cc:511] Master 127.3.115.62:40765 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:02:34.017474 11743 ts_manager.cc:194] Registered new tserver with Master: c687cfe3f2344082ba4a38165ef48a1a (127.3.115.2:35103)
I20260620 06:02:34.018091 11743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:49885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:34.035192 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:40765
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:34.145117 12073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:34.145411 12073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:34.145455 12073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:34.149612 12073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:34.149681 12073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:34.149771 12073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:34.155148 12073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:34.156260 12073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:34.157388 12073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:02:34.167054 12073 server_base.cc:1034] running on GCE node
W20260620 06:02:34.167033 12079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:34.167007 12081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:34.167007 12078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:34.167811 12073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:34.168471 12073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:34.169652 12073 hybrid_clock.cc:648] HybridClock initialized: now 1781935354169633 us; error 40 us; skew 500 ppm
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935354, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:34.172820 12073 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:34.173728 12073 webserver.cc:460] Webserver started at http://127.3.115.3:35975/ using document root <none> and password file <none>
I20260620 06:02:34.174345 12073 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:34.174407 12073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:34.174579 12073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:34.176774 12073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "197d7e1113934ac6aaf55318cc6df494"
format_stamp: "Formatted at 2026-06-20 06:02:34 on dist-test-slave-4h14"
server_key: "5a0a43507f60a6be45156a00403f1989"
server_key_iv: "9d01d51abac8e5d53546261f91dc68fa"
server_key_version: "encryptionkey@0"
I20260620 06:02:34.177299 12073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "197d7e1113934ac6aaf55318cc6df494"
format_stamp: "Formatted at 2026-06-20 06:02:34 on dist-test-slave-4h14"
server_key: "5a0a43507f60a6be45156a00403f1989"
server_key_iv: "9d01d51abac8e5d53546261f91dc68fa"
server_key_version: "encryptionkey@0"
I20260620 06:02:34.181217 12073 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:34.183964 12088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.185228 12073 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:34.185319 12073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "197d7e1113934ac6aaf55318cc6df494"
format_stamp: "Formatted at 2026-06-20 06:02:34 on dist-test-slave-4h14"
server_key: "5a0a43507f60a6be45156a00403f1989"
server_key_iv: "9d01d51abac8e5d53546261f91dc68fa"
server_key_version: "encryptionkey@0"
I20260620 06:02:34.185406 12073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:34.195267 12073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:34.198221 12073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:34.198369 12073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:34.198863 12073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:34.199604 12073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:34.199649 12073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.199689 12073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:34.199707 12073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.208544 12073 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:37203
I20260620 06:02:34.208566 12201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:37203 every 8 connection(s)
I20260620 06:02:34.209474 12073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:34.211239 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12073
I20260620 06:02:34.211400 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:34.211699 3532 external_mini_cluster.cc:1428] Setting key 7020697a554a8c946f3f402a6a1533a3
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935354, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:34.223768 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.211320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38839 (local address 127.3.115.62:40765)
0620 06:02:34.211501 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.211507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.212538 (+ 1031us) server_negotiation.cc:407] Connection header received
0620 06:02:34.214117 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.214120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.214164 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.214222 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.215351 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.215862 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.217115 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.217342 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.217418 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.220033 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:34.220036 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:34.220038 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:34.220062 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:34.221675 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:34.222216 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:34.222219 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:34.222221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:34.222311 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:34.222687 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:34.222692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:34.222694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:34.222884 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:34.222993 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:34.223486 (+ 493us) server_negotiation.cc:299] Negotiation successful
0620 06:02:34.223602 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260620 06:02:34.224810 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.211785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:38839)
0620 06:02:34.212374 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.212433 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.213883 (+ 1450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.214368 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.214375 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.214727 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.215211 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.215223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.215982 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.215985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.217013 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.217022 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.217469 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.217471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.217609 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.218246 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.218263 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.219862 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:34.221818 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:34.221824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:34.221847 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:34.222100 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:34.222401 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:34.222406 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:34.222408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:34.222594 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:34.222991 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:34.222998 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:34.223256 (+ 258us) client_negotiation.cc:769] Sending connection context
0620 06:02:34.223447 (+ 191us) client_negotiation.cc:240] Negotiation successful
0620 06:02:34.223778 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":140,"threads_started":1}
I20260620 06:02:34.226025 12202 heartbeater.cc:347] Connected to a master server at 127.3.115.62:40765
I20260620 06:02:34.226287 12202 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:34.226794 12202 heartbeater.cc:511] Master 127.3.115.62:40765 requested a full tablet report, sending...
I20260620 06:02:34.227800 11742 ts_manager.cc:194] Registered new tserver with Master: 197d7e1113934ac6aaf55318cc6df494 (127.3.115.3:37203)
I20260620 06:02:34.228402 11742 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:38839
I20260620 06:02:34.239116 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab.
Entry for principal oryx/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab.
WARNING: no policy specified for oryx/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab.
Entry for principal oryx/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab.
WARNING: no policy specified for oryx/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab.
Entry for principal oryx/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab.
I20260620 06:02:34.307300 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11805
I20260620 06:02:34.312891 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11939
I20260620 06:02:34.318344 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12073
I20260620 06:02:34.323616 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 11711
I20260620 06:02:34.330029 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40765
--webserver_interface=127.3.115.62
--webserver_port=36627
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:40765
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:34.439697 12211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:34.439970 12211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:34.440016 12211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:34.443938 12211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:34.443997 12211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:34.444015 12211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:34.444033 12211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:34.444048 12211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:34.449290 12211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:40765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=36627
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:34.450320 12211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:34.451292 12211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:34.459928 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:34.459980 12211 server_base.cc:1034] running on GCE node
W20260620 06:02:34.459940 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:34.459940 12219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:34.460893 12211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:34.462030 12211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:34.463218 12211 hybrid_clock.cc:648] HybridClock initialized: now 1781935354463203 us; error 30 us; skew 500 ppm
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935354, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:34.466028 12211 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:34.466918 12211 webserver.cc:460] Webserver started at http://127.3.115.62:36627/ using document root <none> and password file <none>
I20260620 06:02:34.467427 12211 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:34.467497 12211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:34.471236 12211 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:34.473337 12226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.474561 12211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:34.474656 12211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "0333b9f4b165746070923e76ae3388c4"
server_key_iv: "b1a519255ee7e72c85d3fa640acd8a31"
server_key_version: "encryptionkey@0"
I20260620 06:02:34.475164 12211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:34.493891 12211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:34.496595 12211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:34.496815 12211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:34.503844 12211 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:40765
I20260620 06:02:34.503857 12278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:40765 every 8 connection(s)
I20260620 06:02:34.504976 12211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:34.506198 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12211
I20260620 06:02:34.506611 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:42019
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:40765
--webserver_port=40897
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:02:34.509487 12279 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:02:34.512050 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Bootstrap starting.
I20260620 06:02:34.534686 12279 log.cc:824] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:34.542889 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:02:34.543313 12279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Bootstrap complete.
I20260620 06:02:34.548816 12279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:34.549324 12279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23b2d8fb0b4d48f88c554e5d9f1b8e89, State: Initialized, Role: FOLLOWER
I20260620 06:02:34.549891 12279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:34.550047 12279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:34.550119 12279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:34.550210 12279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:02:34.552790 12279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:34.553159 12279 leader_election.cc:302] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23b2d8fb0b4d48f88c554e5d9f1b8e89; no voters:
I20260620 06:02:34.553508 12279 leader_election.cc:288] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:02:34.553651 12283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:02:34.553921 12283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [term 2 LEADER]: Becoming Leader. State: Replica: 23b2d8fb0b4d48f88c554e5d9f1b8e89, State: Running, Role: LEADER
I20260620 06:02:34.554230 12283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } }
I20260620 06:02:34.554838 12279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:34.555874 12285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23b2d8fb0b4d48f88c554e5d9f1b8e89. Latest consensus state: current_term: 2 leader_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } } }
I20260620 06:02:34.556018 12285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:34.555871 12284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23b2d8fb0b4d48f88c554e5d9f1b8e89" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40765 } } }
I20260620 06:02:34.556113 12284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:34.556391 12292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:34.559952 12292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:34.560643 12292 catalog_manager.cc:1259] Loaded cluster ID: 7d26e17e26474e6c8662e279422ec54b
I20260620 06:02:34.560736 12292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:34.562476 12292 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:34.563033 12292 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 23b2d8fb0b4d48f88c554e5d9f1b8e89: Loaded TSK: 0
I20260620 06:02:34.563957 12292 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:02:34.641306 12281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:34.641618 12281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:34.641695 12281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:34.645905 12281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:34.645998 12281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:34.646093 12281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:34.651386 12281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.1.keytab
--principal=oryx/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:42019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=40897
--tserver_master_addrs=127.3.115.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:34.652455 12281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:34.653553 12281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:34.663615 12306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:34.663578 12281 server_base.cc:1034] running on GCE node
W20260620 06:02:34.663570 12305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:34.663578 12308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:34.664220 12281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:34.665026 12281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:34.666217 12281 hybrid_clock.cc:648] HybridClock initialized: now 1781935354666195 us; error 46 us; skew 500 ppm
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935354, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:34.669253 12281 init.cc:377] Logged in from keytab as oryx/127.3.115.1@KRBTEST.COM (short username oryx)
I20260620 06:02:34.670300 12281 webserver.cc:460] Webserver started at http://127.3.115.1:40897/ using document root <none> and password file <none>
I20260620 06:02:34.670885 12281 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:34.670992 12281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:34.675285 12281 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:02:34.677428 12315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.678809 12281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:34.678927 12281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "2a607b2307b24f38b7cfeb018a514f65"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "7bc6ccdcd74fefa5cedc689174e94b50"
server_key_iv: "3e5b9f1147d6f084d1de384c3c022cd6"
server_key_version: "encryptionkey@0"
I20260620 06:02:34.679557 12281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:34.690802 12281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:34.694227 12281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:34.694434 12281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:34.695034 12281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:34.695850 12281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:34.695896 12281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.695958 12281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:34.695989 12281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.704130 12281 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:42019
I20260620 06:02:34.704308 12428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:42019 every 8 connection(s)
I20260620 06:02:34.705273 12281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:34.713771 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12281
I20260620 06:02:34.714218 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:35103
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:40765
--webserver_port=32777
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:34.719412 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.707677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:39219)
0620 06:02:34.708185 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.708239 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.709668 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.710907 (+ 1239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.710914 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.711276 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.711731 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.711753 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.713095 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.713098 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.714164 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.714181 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.715227 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.715230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.715378 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.716030 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.716057 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.718666 (+ 2609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":156,"threads_started":1}
W20260620 06:02:34.719895 12429 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:40765 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.115.62:40765: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
W20260620 06:02:34.719933 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:34.707559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:39219 (local address 127.3.115.62:40765)
0620 06:02:34.708055 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.708074 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.708369 (+ 295us) server_negotiation.cc:407] Connection header received
0620 06:02:34.710108 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.710120 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.710452 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.710756 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.711903 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.712974 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.714288 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.715075 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.715259 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.718976 (+ 3717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:39219: BlockingRecv error: recv got EOF from 127.3.115.1:39219 (error 108)
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":125,"threads_started":1}
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:34.725423 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.720513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:49419)
0620 06:02:34.720605 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.720621 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.720763 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.721008 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.721011 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.721168 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.721252 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.721257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.721965 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.721971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.722508 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.722515 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.722947 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.722950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.723045 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.723529 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.723541 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.725286 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260620 06:02:34.725512 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:34.720607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:49419 (local address 127.3.115.62:40765)
0620 06:02:34.720748 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.720752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.720765 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:02:34.720859 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.720861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.720912 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.720997 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.721361 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.721863 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.722622 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.722841 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.722947 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.725399 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:49419: BlockingRecv error: recv got EOF from 127.3.115.1:49419 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:34.731055 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.725941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:34263)
0620 06:02:34.726047 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.726062 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.726131 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.726379 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.726381 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.726498 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.726562 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.726566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.727531 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.727533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.728039 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.728045 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.728595 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.728597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.728652 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.729164 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.729178 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.730932 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:02:34.731174 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:34.725977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:34263 (local address 127.3.115.62:40765)
0620 06:02:34.726078 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.726082 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.726101 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:02:34.726222 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.726225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.726266 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.726359 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.726659 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.727431 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.728143 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.728503 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.728596 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.731048 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:34263: BlockingRecv error: recv got EOF from 127.3.115.1:34263 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 06:02:34.731438 12429 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:02:34.829509 12433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:34.829797 12433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:34.829842 12433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:34.833961 12433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:34.834026 12433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:34.834085 12433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:34.839363 12433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.2.keytab
--principal=oryx/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:35103
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=32777
--tserver_master_addrs=127.3.115.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:34.840441 12433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:34.841454 12433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:34.850819 12441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:34.850831 12438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:34.850826 12439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:34.851200 12433 server_base.cc:1034] running on GCE node
I20260620 06:02:34.851600 12433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:34.852227 12433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:34.853446 12433 hybrid_clock.cc:648] HybridClock initialized: now 1781935354853431 us; error 28 us; skew 500 ppm
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935354, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:34.856431 12433 init.cc:377] Logged in from keytab as oryx/127.3.115.2@KRBTEST.COM (short username oryx)
I20260620 06:02:34.857342 12433 webserver.cc:460] Webserver started at http://127.3.115.2:32777/ using document root <none> and password file <none>
I20260620 06:02:34.857909 12433 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:34.858017 12433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:34.861855 12433 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.003s
I20260620 06:02:34.864705 12448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.866591 12433 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:02:34.866727 12433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "c687cfe3f2344082ba4a38165ef48a1a"
format_stamp: "Formatted at 2026-06-20 06:02:33 on dist-test-slave-4h14"
server_key: "00f622ec51f952c3675259e4e8bdc148"
server_key_iv: "be146b8a47441acd2c74dfce99cdcdab"
server_key_version: "encryptionkey@0"
I20260620 06:02:34.867307 12433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:34.883490 12433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:34.886818 12433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:34.887030 12433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:34.887682 12433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:34.888543 12433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:34.888589 12433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.888629 12433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:34.888720 12433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:34.898365 12433 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:35103
I20260620 06:02:34.898381 12561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:35103 every 8 connection(s)
I20260620 06:02:34.899490 12433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:34.901288 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12433
I20260620 06:02:34.901845 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:37203
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:40765
--webserver_port=35975
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:02:34.913729 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:34.901533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:39253 (local address 127.3.115.62:40765)
0620 06:02:34.901722 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.901729 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.902837 (+ 1108us) server_negotiation.cc:407] Connection header received
0620 06:02:34.904420 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.904425 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.904506 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.904623 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.906839 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.907800 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.909168 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.909614 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.909747 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.913593 (+ 3846us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:39253: BlockingRecv error: recv got EOF from 127.3.115.2:39253 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:02:34.914358 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.902020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:39253)
0620 06:02:34.902609 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.902676 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.904110 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.904959 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.904968 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.905727 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.906634 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.906654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.907969 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.907972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.909034 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.909044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.909807 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.909811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.909987 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.910690 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.910708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.913419 (+ 2711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":117,"threads_started":1}
W20260620 06:02:34.914916 12562 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:40765 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.115.62:40765: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:34.920914 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.915434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:34909)
0620 06:02:34.915558 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.915575 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.915657 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.915902 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.915904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.916036 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.916180 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.916190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.917221 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.917224 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.917764 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.917771 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.918326 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.918328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.918396 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.918900 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.918911 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.920814 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 06:02:34.921082 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:34.915514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:34909 (local address 127.3.115.62:40765)
0620 06:02:34.915635 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.915640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.915654 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:34.915746 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.915749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.915795 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.915880 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.916295 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.917120 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.917866 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.918212 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.918311 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.920969 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:34909: BlockingRecv error: recv got EOF from 127.3.115.2:34909 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:34 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:34.926725 12564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:34.921594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:39337)
0620 06:02:34.921720 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:34.921734 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:34.921810 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:34.922033 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:34.922036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:34.922193 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:34.922282 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.922287 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.922950 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.922955 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.923812 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:34.923822 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.924234 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.924239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:34.924322 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.924856 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:34.924869 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:34.926635 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 06:02:34.926895 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:34.921704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:39337 (local address 127.3.115.62:40765)
0620 06:02:34.921804 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:34.921807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:34.921820 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:02:34.921894 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:34.921896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:34.921930 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:34.921992 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:34.922386 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.922858 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.923930 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:34.924127 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:34.924206 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:34.926783 (+ 2577us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:39337: BlockingRecv error: recv got EOF from 127.3.115.2:39337 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 06:02:34.927327 12562 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:02:35.019568 12565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:35.019826 12565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:35.019869 12565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:35.023898 12565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:35.023964 12565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:35.024057 12565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:35.029433 12565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/krb5kdc/oryx_127.3.115.3.keytab
--principal=oryx/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:37203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=35975
--tserver_master_addrs=127.3.115.62:40765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:35.030575 12565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:35.031641 12565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:35.040920 12570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:35.040978 12571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:35.040923 12573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:35.041137 12565 server_base.cc:1034] running on GCE node
I20260620 06:02:35.041610 12565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:35.042239 12565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:35.043431 12565 hybrid_clock.cc:648] HybridClock initialized: now 1781935355043408 us; error 53 us; skew 500 ppm
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935355, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:35.046166 12565 init.cc:377] Logged in from keytab as oryx/127.3.115.3@KRBTEST.COM (short username oryx)
I20260620 06:02:35.047209 12565 webserver.cc:460] Webserver started at http://127.3.115.3:35975/ using document root <none> and password file <none>
I20260620 06:02:35.047788 12565 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:35.047861 12565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:35.051553 12565 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:35.053568 12580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:35.054828 12565 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:02:35.054992 12565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "197d7e1113934ac6aaf55318cc6df494"
format_stamp: "Formatted at 2026-06-20 06:02:34 on dist-test-slave-4h14"
server_key: "5a0a43507f60a6be45156a00403f1989"
server_key_iv: "9d01d51abac8e5d53546261f91dc68fa"
server_key_version: "encryptionkey@0"
I20260620 06:02:35.055528 12565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:35.081563 12565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:35.084740 12565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:35.084897 12565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:35.085439 12565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:35.086196 12565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:35.086242 12565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:35.086282 12565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:35.086302 12565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:35.095144 12565 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:37203
I20260620 06:02:35.095170 12693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:37203 every 8 connection(s)
I20260620 06:02:35.096117 12565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:35.099311 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12565
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 06:02:35.107436 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:35.097831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:39561 (local address 127.3.115.62:40765)
0620 06:02:35.097962 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:35.097965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:35.098760 (+ 795us) server_negotiation.cc:407] Connection header received
0620 06:02:35.100040 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:35.100044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:35.100102 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:35.100181 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:35.101565 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.102208 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.103481 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.103866 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.103946 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.107332 (+ 3386us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:39561: BlockingRecv error: recv got EOF from 127.3.115.3:39561 (error 108)
Metrics: {"mutex_wait_us":103,"server-negotiator.queue_time_us":45}
I20260620 06:02:35.108260 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:35.098167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:39561)
0620 06:02:35.098601 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:35.098643 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:35.099824 (+ 1181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:35.100352 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:35.100359 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:35.100913 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:35.101396 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.101410 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.102338 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.102340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.103339 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.103349 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.104026 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.104033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.104235 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.104800 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:35.104815 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:35.107226 (+ 2411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
W20260620 06:02:35.108793 12694 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:40765 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.115.62:40765: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
I20260620 06:02:35.109532 12698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:35.100309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.3.115.62:40765)
0620 06:02:35.100559 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:35.100563 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:35.100576 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:02:35.100616 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:35.100619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:35.100687 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:35.100783 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:35.101533 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.102139 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.103065 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.103383 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.103483 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.105134 (+ 1651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:35.105141 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:35.105158 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:35.105187 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:35.107083 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:35.107700 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:35.107706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:35.107710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:35.107826 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:35.108113 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:35.108115 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:35.108117 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:35.108517 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:35.108761 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:35.109104 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 06:02:35.109344 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":168,"thread_start_us":71,"threads_started":1}
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:35.114609 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:35.109251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:51625)
0620 06:02:35.109355 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:35.109369 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:35.109443 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:35.109761 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:35.109764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:35.109916 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:35.110019 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.110026 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.110924 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.110927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.111685 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.111694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.112220 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.112227 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.112380 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.112917 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:35.112933 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:35.114525 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 06:02:35.114735 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:35.109418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:51625 (local address 127.3.115.62:40765)
0620 06:02:35.109525 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:35.109529 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:35.109547 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:02:35.109600 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:35.109604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:35.109659 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:35.109738 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:35.110127 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.110793 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.111814 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.112086 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.112192 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.114633 (+ 2441us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:51625: BlockingRecv error: recv got EOF from 127.3.115.3:51625 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:35.119511 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:35.115206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:60107)
0620 06:02:35.115351 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:35.115363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:35.115423 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:35.115623 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:35.115624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:35.115738 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:35.115805 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.115810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.116425 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.116427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.116975 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.116983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.117371 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.117373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.117427 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.117882 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:35.117893 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:35.119432 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:02:35.119647 12431 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:35.115328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:60107 (local address 127.3.115.62:40765)
0620 06:02:35.115465 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:35.115468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:35.115479 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:02:35.115505 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:35.115507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:35.115537 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:35.115598 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:35.115894 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.116344 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.117076 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.117281 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.117357 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.119543 (+ 2186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:60107: BlockingRecv error: recv got EOF from 127.3.115.3:60107 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260620 06:02:35.119832 12694 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:35.737941 12699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:35.732078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:38339)
0620 06:02:35.732372 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:35.732386 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:35.732483 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:35.732808 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:35.732810 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:35.733012 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:35.733128 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.733134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.733906 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.733909 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.734596 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.734605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.735232 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.735236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.735353 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.735940 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:35.735954 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:35.737806 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":84,"threads_started":1}
W20260620 06:02:35.738142 12700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:35.732208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:38339 (local address 127.3.115.62:40765)
0620 06:02:35.732454 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:35.732458 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:35.732472 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:35.732586 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:35.732589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:35.732634 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:35.732777 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:35.733240 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.733805 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.734733 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.735100 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.735216 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.737966 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:38339: BlockingRecv error: recv got EOF from 127.3.115.1:38339 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":96,"threads_started":1}
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:35 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:35.933622 12701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:35.927945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:34527)
0620 06:02:35.928284 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:35.928308 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:35.928450 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:35.928776 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:35.928778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:35.929006 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:35.929102 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.929107 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.929869 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.929875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.930466 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:35.930474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.930930 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.930934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:35.931019 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.931597 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:35.931609 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:35.933439 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":83,"threads_started":1}
W20260620 06:02:35.933660 12700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:35.928076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:34527 (local address 127.3.115.62:40765)
0620 06:02:35.928247 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:35.928252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:35.928387 (+ 135us) server_negotiation.cc:407] Connection header received
0620 06:02:35.928551 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:35.928556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:35.928613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:35.928753 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:35.929226 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.929754 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.930590 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:35.930817 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:35.930926 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:35.933559 (+ 2633us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:34527: BlockingRecv error: recv got EOF from 127.3.115.2:34527 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 20 06:02:36 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:36 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:36.126295 12702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:36.120607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:51621)
0620 06:02:36.120937 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:36.120950 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:36.121092 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:36.121418 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:36.121421 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:36.121657 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:36.121781 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:36.121790 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.122544 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.122548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:36.123139 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:36.123147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.123601 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.123604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:36.123690 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:36.124198 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:36.124210 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:36.126119 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":102,"threads_started":1}
W20260620 06:02:36.126344 12700 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:36.120714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:51621 (local address 127.3.115.62:40765)
0620 06:02:36.120880 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:36.120885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:36.121054 (+ 169us) server_negotiation.cc:407] Connection header received
0620 06:02:36.121207 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:36.121213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:36.121281 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:36.121384 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:36.121891 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.122426 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.123256 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.123486 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.123596 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:36.126239 (+ 2643us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:51621: BlockingRecv error: recv got EOF from 127.3.115.3:51621 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Jun 20 06:02:36 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:36 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:36.744853 12703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:36.738775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:59931)
0620 06:02:36.739070 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:36.739084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:36.739175 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:36.739537 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:36.739540 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:36.739732 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:36.739837 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:36.739843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.740650 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.740684 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:36.741289 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:36.741297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.741897 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.741901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:36.741986 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:36.742607 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:36.742627 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:36.744638 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
W20260620 06:02:36.744947 12704 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:36.738830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:59931 (local address 127.3.115.62:40765)
0620 06:02:36.739164 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:36.739167 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:36.739182 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:36.739274 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:36.739276 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:36.739320 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:36.739410 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:36.739980 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.740534 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.741455 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.741781 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.741895 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:36.744809 (+ 2914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:59931: BlockingRecv error: recv got EOF from 127.3.115.1:59931 (error 108)
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":187,"threads_started":1}
Jun 20 06:02:36 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:36 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:36.940620 12705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:36.934435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:46101)
0620 06:02:36.934795 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:36.934814 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:36.934972 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:36.935255 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:36.935258 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:36.935534 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:36.935632 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:36.935638 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.936441 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.936446 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:36.937162 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:36.937172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.937772 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.937776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:36.937858 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:36.938490 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:36.938503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:36.940422 (+ 1919us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
W20260620 06:02:36.940712 12704 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:36.934564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:46101 (local address 127.3.115.62:40765)
0620 06:02:36.934765 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:36.934769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:36.934903 (+ 134us) server_negotiation.cc:407] Connection header received
0620 06:02:36.935079 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:36.935083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:36.935139 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:36.935231 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:36.935760 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.936318 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.937283 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:36.937662 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:36.937765 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:36.940565 (+ 2800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:46101: BlockingRecv error: recv got EOF from 127.3.115.2:46101 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 06:02:37 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:37 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:37.132496 12706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:37.127153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:40955)
0620 06:02:37.127480 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:37.127494 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:37.127593 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:37.127830 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:37.127832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:37.128006 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:37.128099 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:37.128103 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.128925 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.128928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:37.129475 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:37.129481 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.129861 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.129864 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:37.129921 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:37.130472 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:37.130483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:37.132272 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":109,"threads_started":1}
W20260620 06:02:37.132505 12704 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:37.127332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:40955 (local address 127.3.115.62:40765)
0620 06:02:37.127490 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:37.127493 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:37.127543 (+ 50us) server_negotiation.cc:407] Connection header received
0620 06:02:37.127677 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:37.127680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:37.127724 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:37.127793 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:37.128195 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.128832 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.129568 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.129769 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.129842 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:37.132412 (+ 2570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:40955: BlockingRecv error: recv got EOF from 127.3.115.3:40955 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 06:02:37 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:37 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:37.751739 12707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:37.745692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:44141)
0620 06:02:37.746004 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:37.746018 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:37.746112 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:37.746377 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:37.746379 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:37.746555 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:37.746654 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:37.746659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.747528 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.747531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:37.748110 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:37.748118 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.748751 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.748756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:37.748866 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:37.749551 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:37.749566 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:37.751623 (+ 2057us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":84,"threads_started":1}
W20260620 06:02:37.751952 12708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:37.745837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:44141 (local address 127.3.115.62:40765)
0620 06:02:37.746084 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:37.746087 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:37.746101 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:37.746205 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:37.746207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:37.746250 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:37.746334 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:37.746832 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.747407 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.748224 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.748557 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.748707 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:37.751787 (+ 3080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:44141: BlockingRecv error: recv got EOF from 127.3.115.1:44141 (error 108)
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
Jun 20 06:02:37 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:37 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:37.946851 12709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:37.941376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:59889)
0620 06:02:37.941627 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:37.941641 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:37.941749 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:37.942015 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:37.942018 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:37.942195 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:37.942293 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:37.942300 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.943091 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.943093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:37.943673 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:37.943681 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.944139 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.944141 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:37.944212 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:37.944823 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:37.944843 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:37.946693 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":92,"threads_started":1}
W20260620 06:02:37.946981 12708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:37.941450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:59889 (local address 127.3.115.62:40765)
0620 06:02:37.941589 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:37.941592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:37.941752 (+ 160us) server_negotiation.cc:407] Connection header received
0620 06:02:37.941818 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:37.941820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:37.941868 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:37.941943 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:37.942428 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.942992 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.943812 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:37.944031 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:37.944121 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:37.946843 (+ 2722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:59889: BlockingRecv error: recv got EOF from 127.3.115.2:59889 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 20 06:02:38 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:38 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:38.139029 12710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:38.133395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:49381)
0620 06:02:38.133676 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:38.133691 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:38.133819 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:38.134053 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:38.134056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:38.134232 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:38.134328 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:38.134334 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.135131 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.135134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:38.135697 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:38.135705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.136178 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.136180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:38.136242 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:38.136850 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:38.136871 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:38.138839 (+ 1968us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":117,"threads_started":1}
W20260620 06:02:38.139094 12708 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:38.133522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:49381 (local address 127.3.115.62:40765)
0620 06:02:38.133657 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:38.133664 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:38.133766 (+ 102us) server_negotiation.cc:407] Connection header received
0620 06:02:38.133889 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:38.133891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:38.133937 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:38.134019 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:38.134467 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.135015 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.135823 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.136061 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.136167 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:38.138978 (+ 2811us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:49381: BlockingRecv error: recv got EOF from 127.3.115.3:49381 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 20 06:02:38 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:38 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:38.762472 12711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:38.752777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:60613)
0620 06:02:38.753052 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:38.753068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:38.753184 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:38.753473 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:38.753475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:38.753661 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:38.753754 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:38.753759 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.754543 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.754546 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:38.755194 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:38.755207 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.755679 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.755681 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:38.755747 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:38.760213 (+ 4466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:38.760226 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:38.762346 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":86,"threads_started":1}
W20260620 06:02:38.762614 12712 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:38.752838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:60613 (local address 127.3.115.62:40765)
0620 06:02:38.753083 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:38.753087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:38.753127 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:02:38.753277 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:38.753279 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:38.753325 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:38.753433 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:38.753881 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.754413 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.755322 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.755555 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.755641 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:38.762473 (+ 6832us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:60613: BlockingRecv error: recv got EOF from 127.3.115.1:60613 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":60,"threads_started":1}
Jun 20 06:02:38 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:38 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:38.953648 12713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:38.947719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:60887)
0620 06:02:38.948173 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:38.948186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:38.948326 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:38.948600 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:38.948603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:38.948840 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:38.948936 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:38.948942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.949726 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.949732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:38.950289 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:38.950297 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.950803 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.950807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:38.950895 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:38.951461 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:38.951473 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:38.953532 (+ 2059us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":84,"threads_started":1}
W20260620 06:02:38.953863 12712 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:38.947864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:60887 (local address 127.3.115.62:40765)
0620 06:02:38.948122 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:38.948126 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:38.948260 (+ 134us) server_negotiation.cc:407] Connection header received
0620 06:02:38.948442 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:38.948446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:38.948492 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:38.948577 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:38.949058 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.949614 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.950416 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:38.950679 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:38.950793 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:38.953713 (+ 2920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:60887: BlockingRecv error: recv got EOF from 127.3.115.2:60887 (error 108)
Metrics: {"server-negotiator.queue_time_us":159}
Jun 20 06:02:39 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:39 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:39.146013 12714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:39.140000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:50833)
0620 06:02:39.140269 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:39.140288 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:39.140430 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:39.140732 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:39.140735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:39.140919 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:39.141017 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:39.141022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.141898 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.141901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:39.142473 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:39.142480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.142932 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.142935 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:39.142998 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:39.143602 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:39.143615 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:39.145840 (+ 2225us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":122,"threads_started":1}
W20260620 06:02:39.146071 12712 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:39.140074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:50833 (local address 127.3.115.62:40765)
0620 06:02:39.140248 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:39.140252 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:39.140400 (+ 148us) server_negotiation.cc:407] Connection header received
0620 06:02:39.140517 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:39.140520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:39.140578 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:39.140678 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:39.141184 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.141779 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.142598 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.142825 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.142910 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:39.145962 (+ 3052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:50833: BlockingRecv error: recv got EOF from 127.3.115.3:50833 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 20 06:02:39 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:39 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:39.768925 12715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:39.763396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:39597)
0620 06:02:39.763689 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:39.763703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:39.763801 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:39.764053 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:39.764056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:39.764252 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:39.764352 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:39.764357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.765205 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.765208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:39.765776 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:39.765784 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.766224 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.766226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:39.766284 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:39.766857 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:39.766869 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:39.768799 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
W20260620 06:02:39.769134 12716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:39.763456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:39597 (local address 127.3.115.62:40765)
0620 06:02:39.763696 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:39.763701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:39.763757 (+ 56us) server_negotiation.cc:407] Connection header received
0620 06:02:39.763902 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:39.763904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:39.763947 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:39.764014 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:39.764463 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.765097 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.765905 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.766128 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.766209 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:39.768949 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:39597: BlockingRecv error: recv got EOF from 127.3.115.1:39597 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":60,"threads_started":1}
Jun 20 06:02:39 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:39 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:39.960341 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:39.954606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:42593)
0620 06:02:39.954903 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:39.954917 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:39.955065 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:39.955344 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:39.955346 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:39.955528 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:39.955619 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:39.955624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.956521 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.956524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:39.957120 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:39.957129 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.957553 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.957556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:39.957615 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:39.958169 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:39.958181 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:39.960169 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
W20260620 06:02:39.960518 12716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:39.954762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:42593 (local address 127.3.115.62:40765)
0620 06:02:39.954908 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:39.954912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:39.955016 (+ 104us) server_negotiation.cc:407] Connection header received
0620 06:02:39.955167 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:39.955170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:39.955227 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:39.955305 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:39.955722 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.956416 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.957217 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:39.957452 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:39.957534 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:39.960369 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:42593: BlockingRecv error: recv got EOF from 127.3.115.2:42593 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 06:02:40 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:40 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:40.153024 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:40.146952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:53405)
0620 06:02:40.147334 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:40.147353 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:40.147512 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:40.147817 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:40.147821 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:40.148042 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:40.148138 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:40.148144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.148988 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.148992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:40.149693 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:40.149710 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.150289 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.150320 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:40.150417 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:40.151000 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:40.151013 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:40.152896 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":103,"threads_started":1}
W20260620 06:02:40.153156 12716 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:40.147116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:53405 (local address 127.3.115.62:40765)
0620 06:02:40.147335 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:40.147341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:40.147453 (+ 112us) server_negotiation.cc:407] Connection header received
0620 06:02:40.147616 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:40.147619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:40.147676 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:40.147769 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:40.148279 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.148877 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.149829 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.150115 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.150252 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:40.153038 (+ 2786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:53405: BlockingRecv error: recv got EOF from 127.3.115.3:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 20 06:02:40 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:40 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:40.775923 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:40.769755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:33385)
0620 06:02:40.770080 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:40.770095 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:40.770190 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:40.770485 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:40.770487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:40.770675 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:40.770766 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:40.770780 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.771718 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.771721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:40.772351 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:40.772360 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.772858 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.772860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:40.772986 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:40.773608 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:40.773624 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:40.775740 (+ 2116us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":91,"threads_started":1}
W20260620 06:02:40.775979 12720 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:40.769893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:33385 (local address 127.3.115.62:40765)
0620 06:02:40.770139 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:40.770143 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:40.770157 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:40.770313 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:40.770315 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:40.770361 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:40.770441 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:40.770985 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.771598 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.772469 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.772759 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.772846 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:40.775864 (+ 3018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:33385: BlockingRecv error: recv got EOF from 127.3.115.1:33385 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":60,"threads_started":1}
Jun 20 06:02:40 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:40 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:40.967032 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:40.961172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:50501)
0620 06:02:40.961515 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:40.961532 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:40.961679 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:40.962027 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:40.962030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:40.962318 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:40.962441 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:40.962449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.963355 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.963358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:40.963915 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:40.963922 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.964424 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.964427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:40.964496 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:40.965035 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:40.965047 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:40.966917 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":84,"threads_started":1}
W20260620 06:02:40.967182 12720 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:40.961315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:50501 (local address 127.3.115.62:40765)
0620 06:02:40.961496 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:40.961501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:40.961786 (+ 285us) server_negotiation.cc:407] Connection header received
0620 06:02:40.961841 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:40.961843 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:40.961887 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:40.961985 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:40.962576 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.963248 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.964028 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:40.964298 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:40.964389 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:40.967063 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:50501: BlockingRecv error: recv got EOF from 127.3.115.2:50501 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 06:02:41 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:41 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:41.160168 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:41.153818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:55429)
0620 06:02:41.154140 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:41.154161 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:41.154296 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:41.154583 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:41.154586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:41.154824 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:41.154918 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:41.154924 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.155808 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.155813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:41.156564 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:41.156574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.157392 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.157395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:41.157514 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:41.158069 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:41.158083 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:41.159997 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":93,"threads_started":1}
W20260620 06:02:41.160235 12720 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:41.153930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:55429 (local address 127.3.115.62:40765)
0620 06:02:41.154094 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:41.154099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:41.154232 (+ 133us) server_negotiation.cc:407] Connection header received
0620 06:02:41.154420 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:41.154423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:41.154470 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:41.154561 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:41.155087 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.155688 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.156971 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.157209 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.157470 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:41.160128 (+ 2658us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:55429: BlockingRecv error: recv got EOF from 127.3.115.3:55429 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 06:02:41 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:41 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:41.782287 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:41.776722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:35449)
0620 06:02:41.777039 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:41.777055 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:41.777181 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:41.777452 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:41.777455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:41.777636 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:41.777727 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:41.777732 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.778571 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.778575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:41.779127 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:41.779134 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.779638 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.779641 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:41.779702 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:41.780269 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:41.780282 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:41.782168 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":111,"threads_started":1}
W20260620 06:02:41.782500 12724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:41.776802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:35449 (local address 127.3.115.62:40765)
0620 06:02:41.777059 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:41.777064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:41.777143 (+ 79us) server_negotiation.cc:407] Connection header received
0620 06:02:41.777273 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:41.777276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:41.777322 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:41.777434 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:41.777823 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.778477 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.779222 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.779543 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.779621 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:41.782296 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:35449: BlockingRecv error: recv got EOF from 127.3.115.1:35449 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":105,"threads_started":1}
Jun 20 06:02:41 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:41 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:41.973569 12725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:41.967775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:36151)
0620 06:02:41.968076 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:41.968091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:41.968192 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:41.968463 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:41.968465 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:41.968646 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:41.968801 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:41.968807 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.969602 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.969604 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:41.970264 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:41.970274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.970742 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.970744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:41.970805 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:41.971434 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:41.971445 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:41.973451 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":97,"threads_started":1}
W20260620 06:02:41.973729 12724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:41.967876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:36151 (local address 127.3.115.62:40765)
0620 06:02:41.968015 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:41.968019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:41.968192 (+ 173us) server_negotiation.cc:407] Connection header received
0620 06:02:41.968298 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:41.968301 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:41.968350 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:41.968425 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:41.968943 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.969503 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.970374 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:41.970635 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:41.970728 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:41.973594 (+ 2866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:36151: BlockingRecv error: recv got EOF from 127.3.115.2:36151 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 06:02:42 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:42 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:42.166739 12726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:42.161074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:48235)
0620 06:02:42.161391 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:42.161405 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:42.161521 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:42.161773 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:42.161775 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:42.161976 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:42.162067 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:42.162072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.162834 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.162851 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:42.163397 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:42.163404 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.163827 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.163829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:42.163901 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:42.164495 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:42.164507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:42.166563 (+ 2056us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":97,"threads_started":1}
W20260620 06:02:42.166833 12724 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:42.161146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:48235 (local address 127.3.115.62:40765)
0620 06:02:42.161293 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:42.161297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:42.161502 (+ 205us) server_negotiation.cc:407] Connection header received
0620 06:02:42.161595 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:42.161598 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:42.161663 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:42.161739 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:42.162194 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.162725 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.163500 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.163729 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.163814 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:42.166713 (+ 2899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:48235: BlockingRecv error: recv got EOF from 127.3.115.3:48235 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 06:02:42 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:42 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:42.789068 12727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:42.783334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:40725)
0620 06:02:42.783613 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:42.783648 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:42.783760 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:42.784108 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:42.784112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:42.784366 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:42.784507 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:42.784514 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.785385 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.785387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:42.785957 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:42.785965 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.786384 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.786387 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:42.786449 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:42.786928 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:42.786940 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:42.788894 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":92,"threads_started":1}
W20260620 06:02:42.789139 12728 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:42.783585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:40725 (local address 127.3.115.62:40765)
0620 06:02:42.783825 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:42.783838 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:42.783852 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:42.783897 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:42.783901 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:42.783956 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:42.784049 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:42.784631 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.785275 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.786068 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.786284 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.786368 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:42.789015 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:40725: BlockingRecv error: recv got EOF from 127.3.115.1:40725 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":93,"threads_started":1}
Jun 20 06:02:42 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:42 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:42.979835 12729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:42.974348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:33759)
0620 06:02:42.974639 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:42.974654 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:42.974750 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:42.974994 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:42.974997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:42.975175 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:42.975269 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:42.975274 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.976146 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.976148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:42.976752 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:42.976761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.977183 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.977186 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:42.977247 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:42.977812 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:42.977824 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:42.979718 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":92,"threads_started":1}
W20260620 06:02:42.980038 12728 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:42.974428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:33759 (local address 127.3.115.62:40765)
0620 06:02:42.974578 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:42.974582 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:42.974724 (+ 142us) server_negotiation.cc:407] Connection header received
0620 06:02:42.974833 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:42.974835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:42.974881 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:42.974951 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:42.975364 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.976046 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.976856 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:42.977074 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:42.977151 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:42.979865 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:33759: BlockingRecv error: recv got EOF from 127.3.115.2:33759 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 06:02:43 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:43 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:43.173290 12730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:43.167741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:36539)
0620 06:02:43.168008 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:43.168022 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:43.168123 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:43.168363 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:43.168365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:43.168541 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:43.168640 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:43.168644 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.169400 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.169402 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:43.169957 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:43.169964 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.170411 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.170414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:43.170478 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:43.171059 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:43.171072 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:43.173120 (+ 2048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":92,"threads_started":1}
W20260620 06:02:43.173624 12728 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:43.167807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:36539 (local address 127.3.115.62:40765)
0620 06:02:43.167972 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:43.167976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:43.168093 (+ 117us) server_negotiation.cc:407] Connection header received
0620 06:02:43.168201 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:43.168203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:43.168249 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:43.168321 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:43.168768 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.169298 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.170054 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.170264 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.170401 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:43.173296 (+ 2895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:36539: BlockingRecv error: recv got EOF from 127.3.115.3:36539 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Jun 20 06:02:43 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:43 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:43.799377 12731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:43.789838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:58419)
0620 06:02:43.790155 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:43.790170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:43.790264 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:43.790567 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:43.790570 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:43.790752 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:43.790863 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:43.790869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.791699 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.791702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:43.792275 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:43.792283 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.792790 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.792793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:43.792870 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:43.796976 (+ 4106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:43.796989 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:43.799133 (+ 2144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":90,"threads_started":1}
W20260620 06:02:43.799484 12732 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:43.789968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:58419 (local address 127.3.115.62:40765)
0620 06:02:43.790206 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:43.790209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:43.790223 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:43.790365 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:43.790368 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:43.790415 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:43.790497 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:43.791005 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.791579 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.792394 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.792617 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.792731 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:43.799311 (+ 6580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:58419: BlockingRecv error: recv got EOF from 127.3.115.1:58419 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":72,"threads_started":1}
Jun 20 06:02:43 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:43 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:43.986641 12733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:43.980637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:49695)
0620 06:02:43.981006 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:43.981020 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:43.981118 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:43.981399 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:43.981402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:43.981585 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:43.981682 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:43.981687 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.982449 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.982451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:43.983125 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:43.983134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.983611 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.983614 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:43.983685 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:43.984437 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:43.984449 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:43.986527 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":108,"threads_started":1}
W20260620 06:02:43.986828 12732 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:43.980746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:49695 (local address 127.3.115.62:40765)
0620 06:02:43.980915 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:43.980919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:43.981092 (+ 173us) server_negotiation.cc:407] Connection header received
0620 06:02:43.981197 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:43.981200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:43.981245 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:43.981359 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:43.981789 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.982335 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.983224 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:43.983481 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:43.983619 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:43.986665 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:49695: BlockingRecv error: recv got EOF from 127.3.115.2:49695 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Jun 20 06:02:44 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:44 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:44.180208 12734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:44.174221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:38291)
0620 06:02:44.174507 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:44.174527 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:44.174674 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:44.174992 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:44.174995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:44.175243 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:44.175374 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:44.175381 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.176350 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.176353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:44.177026 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:44.177035 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.177468 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.177471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:44.177539 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:44.178131 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:44.178146 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:44.180096 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":117,"threads_started":1}
W20260620 06:02:44.180341 12732 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:44.174317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38291 (local address 127.3.115.62:40765)
0620 06:02:44.174463 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:44.174468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:44.174637 (+ 169us) server_negotiation.cc:407] Connection header received
0620 06:02:44.174794 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:44.174797 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:44.174853 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:44.174957 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:44.175519 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.176226 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.177139 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.177355 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.177447 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:44.180226 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:38291: BlockingRecv error: recv got EOF from 127.3.115.3:38291 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 06:02:44 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:44 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:44.805537 12735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:44.800172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:51015)
0620 06:02:44.800444 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:44.800458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:44.800545 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:44.800783 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:44.800785 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:44.800973 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:44.801072 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:44.801077 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.801807 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.801810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:44.802416 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:44.802424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.802817 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.802819 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:44.802882 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:44.803451 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:44.803463 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:44.805422 (+ 1959us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260620 06:02:44.805687 12736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:44.800257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:51015 (local address 127.3.115.62:40765)
0620 06:02:44.800481 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:44.800484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:44.800499 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:02:44.800636 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:44.800639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:44.800699 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:44.800774 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:44.801172 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.801711 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.802515 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.802726 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.802800 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:44.805556 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:51015: BlockingRecv error: recv got EOF from 127.3.115.1:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":54,"threads_started":1}
Jun 20 06:02:44 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:44 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:44.993332 12737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:44.987469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:56531)
0620 06:02:44.987940 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:44.987953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:44.988095 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:44.988419 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:44.988422 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:44.988633 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:44.988753 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:44.988758 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.989533 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.989540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:44.990113 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:44.990121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.990597 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.990601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:44.990687 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:44.991270 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:44.991282 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:44.993134 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":88,"threads_started":1}
W20260620 06:02:44.993376 12736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:44.987589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:56531 (local address 127.3.115.62:40765)
0620 06:02:44.987871 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:44.987876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:44.988037 (+ 161us) server_negotiation.cc:407] Connection header received
0620 06:02:44.988201 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:44.988204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:44.988259 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:44.988395 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:44.988877 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.989423 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.990239 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:44.990482 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:44.990593 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:44.993262 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:56531: BlockingRecv error: recv got EOF from 127.3.115.2:56531 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Jun 20 06:02:45 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:45 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:45.187050 12738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:45.181087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:39999)
0620 06:02:45.181425 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:45.181444 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:45.181584 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:45.181849 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:45.181852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:45.182066 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:45.182160 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:45.182166 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.183003 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.183009 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:45.183571 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:45.183579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.184090 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.184094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:45.184180 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:45.184772 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:45.184784 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:45.186935 (+ 2151us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
W20260620 06:02:45.187244 12736 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:45.181258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:39999 (local address 127.3.115.62:40765)
0620 06:02:45.181437 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:45.181443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:45.181524 (+ 81us) server_negotiation.cc:407] Connection header received
0620 06:02:45.181696 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:45.181698 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:45.181744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:45.181828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:45.182274 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.182893 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.183693 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.183979 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.184088 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:45.187100 (+ 3012us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:39999: BlockingRecv error: recv got EOF from 127.3.115.3:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 20 06:02:45 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:45 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:45.812561 12739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:45.806529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:55797)
0620 06:02:45.806883 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:45.806902 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:45.807012 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:45.807273 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:45.807275 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:45.807467 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:45.807565 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:45.807570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.808329 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.808331 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:45.808931 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:45.808939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.809357 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.809360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:45.809423 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:45.809925 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:45.809936 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:45.812387 (+ 2451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
W20260620 06:02:45.812826 12740 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:45.806641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:55797 (local address 127.3.115.62:40765)
0620 06:02:45.806935 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:45.806940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:45.806957 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:02:45.807108 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:45.807110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:45.807156 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:45.807236 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:45.807672 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.808219 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.809029 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.809250 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.809330 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:45.812559 (+ 3229us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:55797: BlockingRecv error: recv got EOF from 127.3.115.1:55797 (error 108)
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":55,"threads_started":1}
Jun 20 06:02:45 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:45 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:45.999938 12741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:45.994204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:38547)
0620 06:02:45.994483 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:45.994497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:45.994608 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:45.994851 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:45.994854 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:45.995036 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:45.995135 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:45.995140 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.995948 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.995951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:45.996552 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:45.996560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.997078 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.997081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:45.997158 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:45.997705 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:45.997717 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:45.999801 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":67,"threads_started":1}
W20260620 06:02:46.000097 12740 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:45.994341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:38547 (local address 127.3.115.62:40765)
0620 06:02:45.994512 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:45.994516 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:45.994551 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:02:45.994695 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:45.994697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:45.994741 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:45.994810 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:45.995273 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.995820 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.996725 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:45.996961 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:45.997044 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:45.999937 (+ 2893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:38547: BlockingRecv error: recv got EOF from 127.3.115.2:38547 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Jun 20 06:02:46 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:46 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:46.193439 12742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:46.187820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:52493)
0620 06:02:46.188146 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:46.188160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:46.188264 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:46.188544 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:46.188547 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:46.188784 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:46.188903 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:46.188909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.189653 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.189656 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:46.190227 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:46.190235 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.190686 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.190689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:46.190754 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:46.191251 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:46.191262 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:46.193236 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":89,"threads_started":1}
W20260620 06:02:46.193473 12740 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:46.187948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:52493 (local address 127.3.115.62:40765)
0620 06:02:46.188145 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:46.188148 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:46.188255 (+ 107us) server_negotiation.cc:407] Connection header received
0620 06:02:46.188355 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:46.188357 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:46.188405 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:46.188478 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:46.189029 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.189555 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.190336 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.190565 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.190646 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:46.193362 (+ 2716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:52493: BlockingRecv error: recv got EOF from 127.3.115.3:52493 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Jun 20 06:02:46 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:46 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:46.819198 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:46.813329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:47327)
0620 06:02:46.813638 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:46.813652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:46.813744 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:46.814149 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:46.814152 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:46.814376 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:46.814478 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:46.814484 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.815281 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.815284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:46.815851 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:46.815859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.816353 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.816364 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:46.816435 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:46.816999 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:46.817013 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:46.819026 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":71,"threads_started":1}
W20260620 06:02:46.819281 12744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:46.813381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:47327 (local address 127.3.115.62:40765)
0620 06:02:46.813909 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:46.813913 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:46.813923 (+ 10us) server_negotiation.cc:407] Connection header received
0620 06:02:46.813954 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:46.813956 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:46.813995 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:46.814068 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:46.814608 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.815159 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.815982 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:46.816210 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:46.816300 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:46.819154 (+ 2854us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:47327: BlockingRecv error: recv got EOF from 127.3.115.1:47327 (error 108)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":92,"threads_started":1}
Jun 20 06:02:47 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:47 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:47.006863 12745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:47.001102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:36669)
0620 06:02:47.001458 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:47.001473 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:47.001623 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:47.001931 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:47.001934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:47.002148 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:47.002242 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:47.002249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.003063 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.003069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:47.003631 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:47.003638 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.004099 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.004102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:47.004186 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:47.004776 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:47.004789 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:47.006693 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
W20260620 06:02:47.006927 12744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:47.001262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:36669 (local address 127.3.115.62:40765)
0620 06:02:47.001415 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:47.001419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:47.001565 (+ 146us) server_negotiation.cc:407] Connection header received
0620 06:02:47.001728 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:47.001730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:47.001778 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:47.001884 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:47.002388 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.002950 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.003755 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.003982 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.004094 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:47.006824 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:36669: BlockingRecv error: recv got EOF from 127.3.115.2:36669 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 20 06:02:47 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:47 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:47.199829 12746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:47.194254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:35593)
0620 06:02:47.194560 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:47.194573 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:47.194705 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:47.194989 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:47.194992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:47.195197 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:47.195288 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:47.195294 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.196059 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.196064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:47.196701 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:47.196709 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.197181 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.197185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:47.197270 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:47.197792 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:47.197803 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:47.199658 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":101,"threads_started":1}
W20260620 06:02:47.199893 12744 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:47.194327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:35593 (local address 127.3.115.62:40765)
0620 06:02:47.194502 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:47.194507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:47.194645 (+ 138us) server_negotiation.cc:407] Connection header received
0620 06:02:47.194812 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:47.194816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:47.194870 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:47.194967 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:47.195408 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.195947 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.196822 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.197067 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.197178 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:47.199777 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:35593: BlockingRecv error: recv got EOF from 127.3.115.3:35593 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 06:02:47 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:47 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:47.825661 12747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:47.820135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:44775)
0620 06:02:47.820472 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:47.820492 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:47.820602 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:47.820937 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:47.820939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:47.821120 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:47.821214 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:47.821220 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.821994 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.821997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:47.822575 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:47.822583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.823008 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.823010 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:47.823073 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:47.823578 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:47.823590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:47.825534 (+ 1944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
W20260620 06:02:47.825810 12748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:47.820180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:44775 (local address 127.3.115.62:40765)
0620 06:02:47.820469 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:47.820490 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:47.820558 (+ 68us) server_negotiation.cc:407] Connection header received
0620 06:02:47.820750 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:47.820753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:47.820811 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:47.820895 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:47.821311 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.821865 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.822687 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:47.822908 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:47.822989 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:47.825674 (+ 2685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:44775: BlockingRecv error: recv got EOF from 127.3.115.1:44775 (error 108)
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":73,"threads_started":1}
Jun 20 06:02:48 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:48 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:48.013504 12749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:48.007665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:43893)
0620 06:02:48.008013 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:48.008033 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:48.008150 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:48.008501 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:48.008504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:48.008741 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:48.008838 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:48.008844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.009592 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.009595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:48.010177 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:48.010186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.010645 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.010648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:48.010711 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:48.011228 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:48.011241 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:48.013383 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":112,"threads_started":1}
W20260620 06:02:48.013839 12748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:48.007842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:43893 (local address 127.3.115.62:40765)
0620 06:02:48.008014 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:48.008018 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:48.008091 (+ 73us) server_negotiation.cc:407] Connection header received
0620 06:02:48.008273 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:48.008275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:48.008338 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:48.008470 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:48.008947 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.009473 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.010297 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.010546 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.010627 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:48.013553 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:43893: BlockingRecv error: recv got EOF from 127.3.115.2:43893 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Jun 20 06:02:48 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:48 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:48.206465 12750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:48.200930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:44981)
0620 06:02:48.201231 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:48.201244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:48.201384 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:48.201704 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:48.201707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:48.201904 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:48.202013 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:48.202018 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.202787 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.202790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:48.203349 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:48.203357 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.203801 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.203803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:48.203865 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:48.204376 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:48.204388 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:48.206310 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":125,"threads_started":1}
W20260620 06:02:48.206758 12748 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:48.201041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:44981 (local address 127.3.115.62:40765)
0620 06:02:48.201177 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:48.201181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:48.201317 (+ 136us) server_negotiation.cc:407] Connection header received
0620 06:02:48.201487 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:48.201490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:48.201539 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:48.201625 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:48.202134 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.202666 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.203487 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.203704 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.203779 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:48.206506 (+ 2727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:44981: BlockingRecv error: recv got EOF from 127.3.115.3:44981 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 20 06:02:48 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:48 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:48.837498 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:48.826661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:34689)
0620 06:02:48.826964 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:48.826978 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:48.827075 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:48.827451 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:48.827453 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:48.827662 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:48.827758 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:48.827764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.828552 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.828555 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:48.829248 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:48.829258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.829911 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.829914 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:48.829995 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:48.835202 (+ 5207us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:48.835218 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:48.837335 (+ 2117us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
W20260620 06:02:48.837726 12752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:48.826723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:34689 (local address 127.3.115.62:40765)
0620 06:02:48.827020 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:48.827024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:48.827036 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:02:48.827238 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:48.827241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:48.827290 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:48.827392 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:48.827870 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.828430 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.829401 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:48.829783 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:48.829885 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:48.837544 (+ 7659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:34689: BlockingRecv error: recv got EOF from 127.3.115.1:34689 (error 108)
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":58,"threads_started":1}
Jun 20 06:02:49 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:49 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:49.020359 12753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:49.014428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:59079)
0620 06:02:49.014716 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:49.014732 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:49.015008 (+ 276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:49.015298 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:49.015300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:49.015509 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:49.015601 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:49.015607 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.016579 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.016582 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:49.017257 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:49.017266 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.017731 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.017733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:49.017795 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:49.018286 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:49.018299 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:49.020235 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":109,"threads_started":1}
W20260620 06:02:49.020567 12752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:49.014472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:59079 (local address 127.3.115.62:40765)
0620 06:02:49.014652 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:49.014656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:49.014800 (+ 144us) server_negotiation.cc:407] Connection header received
0620 06:02:49.015127 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:49.015131 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:49.015188 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:49.015276 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:49.015760 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.016479 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.017411 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.017632 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.017715 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:49.020427 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:59079: BlockingRecv error: recv got EOF from 127.3.115.2:59079 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 06:02:49 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:49 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.3@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:49.212978 12754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:49.207241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.3:38765)
0620 06:02:49.207545 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:49.207559 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:49.207660 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:49.207900 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:49.207902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:49.208086 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:49.208179 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:49.208185 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.208984 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.208987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:49.209572 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:49.209580 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.210083 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.210086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:49.210155 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:49.210722 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:49.210734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:49.212847 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
W20260620 06:02:49.213142 12752 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:49.207305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38765 (local address 127.3.115.62:40765)
0620 06:02:49.207510 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:49.207514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:49.207628 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:02:49.207738 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:49.207740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:49.207785 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:49.207852 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:49.208286 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.208868 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.209704 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.209943 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.210037 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:49.212991 (+ 2954us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.3:38765: BlockingRecv error: recv got EOF from 127.3.115.3:38765 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
Jun 20 06:02:49 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:49 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.1@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:49.844113 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:49.838496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.1:59915)
0620 06:02:49.838785 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:49.838799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:49.838896 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:49.839152 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:49.839155 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:49.839338 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:49.839432 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:49.839439 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.840186 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.840189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:49.840760 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:49.840767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.841243 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.841245 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:49.841301 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:49.841782 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:49.841793 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:49.843921 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
W20260620 06:02:49.844512 12756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:49.838556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:59915 (local address 127.3.115.62:40765)
0620 06:02:49.838812 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:49.838816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:49.838879 (+ 63us) server_negotiation.cc:407] Connection header received
0620 06:02:49.838991 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:49.838993 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:49.839037 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:49.839118 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:49.839546 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.840094 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.840895 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:49.841139 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:49.841232 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:49.844192 (+ 2960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.1:59915: BlockingRecv error: recv got EOF from 127.3.115.1:59915 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":62,"threads_started":1}
Jun 20 06:02:50 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 06:02:50 dist-test-slave-4h14 krb5kdc[11698](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.115.2@KRBTEST.COM for oryx/127.3.115.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 06:02:50.027554 12757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.021454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:40765 (local address 127.3.115.2:34347)
0620 06:02:50.021907 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:50.021926 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:50.022098 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:50.022388 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:50.022391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:50.022613 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:50.022703 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.022708 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.023567 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.023570 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.024160 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.024168 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.024694 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.024698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.024795 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.025488 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:50.025500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:50.027422 (+ 1922us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.115.62:40765: Server oryx/127.3.115.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":130,"threads_started":1}
W20260620 06:02:50.027776 12756 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:50.021635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:34347 (local address 127.3.115.62:40765)
0620 06:02:50.021910 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:50.021916 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:50.022052 (+ 136us) server_negotiation.cc:407] Connection header received
0620 06:02:50.022213 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:50.022216 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:50.022271 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:50.022364 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:50.022837 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.023397 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.024281 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.024536 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.024649 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.027657 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.115.2:34347: BlockingRecv error: recv got EOF from 127.3.115.2:34347 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
I20260620 06:02:50.104800 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12281
I20260620 06:02:50.111053 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12433
I20260620 06:02:50.116489 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12565
I20260620 06:02:50.122028 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12211
2026-06-20T06:02:50Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17971 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T06:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:50Z Disabled control of system clock
I20260620 06:02:50.154119 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:33663
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:33663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 06:02:50.265996 12761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:50.266270 12761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:50.266304 12761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:50.270442 12761 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:50.270507 12761 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:50.270524 12761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:50.270540 12761 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:50.270556 12761 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:50.275379 12761 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:33663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:33663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12761
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:50.276516 12761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:50.277941 12761 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:50.287070 12766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.287070 12767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.287147 12769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:50.287357 12761 server_base.cc:1034] running on GCE node
I20260620 06:02:50.288002 12761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:50.289204 12761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:50.290441 12761 hybrid_clock.cc:648] HybridClock initialized: now 1781935370290418 us; error 34 us; skew 500 ppm
I20260620 06:02:50.292567 12761 webserver.cc:460] Webserver started at http://127.3.115.62:32987/ using document root <none> and password file <none>
I20260620 06:02:50.293226 12761 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:50.293292 12761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:50.293529 12761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:50.295684 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "851eb144f3514f91acbbd0530dda3024"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "3373a227a901fe2694a4d95cd8685d3e"
server_key_iv: "c957a04302ee2c28546206b4dd9c7994"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.296281 12761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "851eb144f3514f91acbbd0530dda3024"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "3373a227a901fe2694a4d95cd8685d3e"
server_key_iv: "c957a04302ee2c28546206b4dd9c7994"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.300194 12761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:50.302872 12775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.304243 12761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:50.304446 12761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "851eb144f3514f91acbbd0530dda3024"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "3373a227a901fe2694a4d95cd8685d3e"
server_key_iv: "c957a04302ee2c28546206b4dd9c7994"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.304576 12761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:50.315912 12761 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:50.316606 12761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:50.316856 12761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:50.324100 12761 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:33663
I20260620 06:02:50.324153 12827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:33663 every 8 connection(s)
I20260620 06:02:50.325225 12761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:50.328760 12828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:50.330289 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12761
I20260620 06:02:50.330391 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:50.330703 3532 external_mini_cluster.cc:1428] Setting key 1959880d832bd40cbe8ef376f2427714
I20260620 06:02:50.336133 12828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024: Bootstrap starting.
I20260620 06:02:50.338891 12828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:50.339825 12828 log.cc:824] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:50.342070 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.332269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57394 (local address 127.3.115.62:33663)
0620 06:02:50.333019 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:50.333041 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:50.333086 (+ 45us) server_negotiation.cc:407] Connection header received
0620 06:02:50.333991 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:50.334017 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:50.334547 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:50.335093 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:50.335564 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.337034 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.338003 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.338679 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.338861 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.339864 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:50.339870 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:50.339887 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:50.340396 (+ 509us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:50.340452 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:50.340473 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:50.340553 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:50.340829 (+ 276us) server_negotiation.cc:299] Negotiation successful
0620 06:02:50.341128 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":117,"threads_started":1}
I20260620 06:02:50.342199 12828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024: No bootstrap required, opened a new log
I20260620 06:02:50.346187 12828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851eb144f3514f91acbbd0530dda3024" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 33663 } }
I20260620 06:02:50.346422 12828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:50.346457 12828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 851eb144f3514f91acbbd0530dda3024, State: Initialized, Role: FOLLOWER
I20260620 06:02:50.346877 12828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851eb144f3514f91acbbd0530dda3024" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 33663 } }
I20260620 06:02:50.347002 12828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:50.347054 12828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:50.347144 12828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:50.348058 12828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851eb144f3514f91acbbd0530dda3024" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 33663 } }
I20260620 06:02:50.348358 12828 leader_election.cc:302] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 851eb144f3514f91acbbd0530dda3024; no voters:
I20260620 06:02:50.348631 12828 leader_election.cc:288] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:50.348809 12833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:50.349092 12833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [term 1 LEADER]: Becoming Leader. State: Replica: 851eb144f3514f91acbbd0530dda3024, State: Running, Role: LEADER
I20260620 06:02:50.349515 12833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851eb144f3514f91acbbd0530dda3024" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 33663 } }
I20260620 06:02:50.350065 12828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:50.351464 12834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "851eb144f3514f91acbbd0530dda3024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851eb144f3514f91acbbd0530dda3024" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 33663 } } }
I20260620 06:02:50.351653 12834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:50.351470 12835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 851eb144f3514f91acbbd0530dda3024. Latest consensus state: current_term: 1 leader_uuid: "851eb144f3514f91acbbd0530dda3024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "851eb144f3514f91acbbd0530dda3024" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 33663 } } }
I20260620 06:02:50.351745 12835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:50.352152 12845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:50.355319 12845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:50.361558 12845 catalog_manager.cc:1347] Generated new cluster ID: 355314e808404a75a0cbc27bc65018b4
I20260620 06:02:50.361645 12845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:50.393095 12845 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:50.394397 12845 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:50.410656 12845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 851eb144f3514f91acbbd0530dda3024: Generated new TSK 0
I20260620 06:02:50.411574 12845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:50.415161 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:33663
--builtin_ntp_servers=127.3.115.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 06:02:50.528805 12852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:50.529079 12852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:50.529116 12852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:50.533279 12852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:50.533334 12852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:50.533393 12852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:50.538103 12852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:50.539109 12852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:50.540413 12852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:50.549862 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.549894 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:50.549909 12852 server_base.cc:1034] running on GCE node
W20260620 06:02:50.550073 12857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:50.550541 12852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:50.551213 12852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:50.552410 12852 hybrid_clock.cc:648] HybridClock initialized: now 1781935370552359 us; error 33 us; skew 500 ppm
I20260620 06:02:50.554193 12852 webserver.cc:460] Webserver started at http://127.3.115.1:43461/ using document root <none> and password file <none>
I20260620 06:02:50.554798 12852 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:50.554863 12852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:50.555074 12852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:50.557184 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "d3d9e2542cde4447a8e32c1a22a3e868"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "5cadccc20450c7b599fe0c36511924b0"
server_key_iv: "b94baecb4117075bd6d31c6bf1cb7f90"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.557750 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "d3d9e2542cde4447a8e32c1a22a3e868"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "5cadccc20450c7b599fe0c36511924b0"
server_key_iv: "b94baecb4117075bd6d31c6bf1cb7f90"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.561473 12852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:02:50.563980 12866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.565371 12852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:50.565521 12852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "d3d9e2542cde4447a8e32c1a22a3e868"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "5cadccc20450c7b599fe0c36511924b0"
server_key_iv: "b94baecb4117075bd6d31c6bf1cb7f90"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.565644 12852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:50.580235 12852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:50.581022 12852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:50.581220 12852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:50.581830 12852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:50.582773 12852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:50.582844 12852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.582916 12852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:50.582959 12852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.592247 12852 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:39661
I20260620 06:02:50.592271 12979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:39661 every 8 connection(s)
I20260620 06:02:50.593201 12852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:50.602131 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12852
I20260620 06:02:50.602267 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:50.602632 3532 external_mini_cluster.cc:1428] Setting key 7687e6e82e7aed9fb3d4261c7b330e9a
I20260620 06:02:50.602929 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.595137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:55739 (local address 127.3.115.62:33663)
0620 06:02:50.595279 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:50.595283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:50.596489 (+ 1206us) server_negotiation.cc:407] Connection header received
0620 06:02:50.597973 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:50.597977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:50.598033 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:50.598120 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:50.599072 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.599611 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.600860 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.601089 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.601177 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.602398 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:50.602401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:50.602403 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:50.602418 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:50.602437 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:50.602451 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:50.602512 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:50.602730 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 06:02:50.602812 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:02:50.604117 12982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.595510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:33663 (local address 127.3.115.1:55739)
0620 06:02:50.596276 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:50.596330 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:50.597767 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:50.598304 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:50.598311 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:50.598409 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:50.598933 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.598947 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.599739 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.599741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.600748 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.600758 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.601240 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.601243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.601405 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.602027 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:50.602029 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:50.602089 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:50.602181 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:50.602541 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:50.602551 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:50.602557 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:02:50.602698 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 06:02:50.603035 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":107,"threads_started":1}
I20260620 06:02:50.604820 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:33663
--builtin_ntp_servers=127.3.115.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:02:50.605509 12980 heartbeater.cc:347] Connected to a master server at 127.3.115.62:33663
I20260620 06:02:50.605774 12980 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:50.606443 12980 heartbeater.cc:511] Master 127.3.115.62:33663 requested a full tablet report, sending...
I20260620 06:02:50.608094 12792 ts_manager.cc:194] Registered new tserver with Master: d3d9e2542cde4447a8e32c1a22a3e868 (127.3.115.1:39661)
I20260620 06:02:50.609417 12792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.1:55739
W20260620 06:02:50.719723 12983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:50.719990 12983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:50.720027 12983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:50.724074 12983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:50.724143 12983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:50.724196 12983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:50.729231 12983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:50.730391 12983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:50.731859 12983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:50.741935 12991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.741931 12988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.741945 12989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:50.742340 12983 server_base.cc:1034] running on GCE node
I20260620 06:02:50.742743 12983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:50.743425 12983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:50.744642 12983 hybrid_clock.cc:648] HybridClock initialized: now 1781935370744607 us; error 49 us; skew 500 ppm
I20260620 06:02:50.746747 12983 webserver.cc:460] Webserver started at http://127.3.115.2:35039/ using document root <none> and password file <none>
I20260620 06:02:50.747354 12983 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:50.747454 12983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:50.747690 12983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:50.749917 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "746c74b1a49241d68cf05df9d9e45a07"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "a2c17336a1bace54208bd9f762fbbd44"
server_key_iv: "e17da41e9dc17e9cd2fd0941394f4595"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.750483 12983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "746c74b1a49241d68cf05df9d9e45a07"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "a2c17336a1bace54208bd9f762fbbd44"
server_key_iv: "e17da41e9dc17e9cd2fd0941394f4595"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.754444 12983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:50.757130 12997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.758574 12983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:50.758719 12983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "746c74b1a49241d68cf05df9d9e45a07"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "a2c17336a1bace54208bd9f762fbbd44"
server_key_iv: "e17da41e9dc17e9cd2fd0941394f4595"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.758847 12983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:50.777931 12983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:50.778708 12983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:50.778923 12983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:50.779590 12983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:50.780601 12983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:50.780690 12983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.780771 12983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:50.780825 12983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.789793 12983 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:34995
I20260620 06:02:50.789814 13110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:34995 every 8 connection(s)
I20260620 06:02:50.790738 12983 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:50.792248 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 12983
I20260620 06:02:50.792379 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:50.792785 3532 external_mini_cluster.cc:1428] Setting key 88eb591c8b90e47e0aa1f3dd48d1976e
I20260620 06:02:50.794931 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:33663
--builtin_ntp_servers=127.3.115.20:35735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 06:02:50.802106 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.792685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:49985 (local address 127.3.115.62:33663)
0620 06:02:50.792835 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:50.792841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:50.794088 (+ 1247us) server_negotiation.cc:407] Connection header received
0620 06:02:50.796116 (+ 2028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:50.796120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:50.796164 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:50.796237 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:50.797320 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.797963 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.799885 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.800107 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.800197 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.801609 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:50.801612 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:50.801614 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:50.801626 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:50.801647 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:50.801655 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:50.801720 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:50.801935 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 06:02:50.801993 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:02:50.803447 13113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.793222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:33663 (local address 127.3.115.2:49985)
0620 06:02:50.793926 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:50.794003 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:50.795801 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:50.796439 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:50.796452 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:50.796579 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:50.797167 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.797185 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.798098 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.798105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.799752 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.799767 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.800274 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.800281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.800546 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.801263 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:50.801266 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:50.801329 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:50.801426 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:50.801748 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:50.801761 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:50.801770 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:02:50.801910 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:02:50.802253 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":146,"threads_started":1}
I20260620 06:02:50.804730 13111 heartbeater.cc:347] Connected to a master server at 127.3.115.62:33663
I20260620 06:02:50.804997 13111 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:50.805538 13111 heartbeater.cc:511] Master 127.3.115.62:33663 requested a full tablet report, sending...
I20260620 06:02:50.806583 12792 ts_manager.cc:194] Registered new tserver with Master: 746c74b1a49241d68cf05df9d9e45a07 (127.3.115.2:34995)
I20260620 06:02:50.807312 12792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.2:49985
W20260620 06:02:50.911047 13114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:50.911321 13114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:50.911363 13114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:50.915436 13114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:50.915498 13114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:50.915553 13114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:50.920400 13114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:33663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:50.921795 13114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:50.923588 13114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:50.934090 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.934082 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:50.934087 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:50.934521 13114 server_base.cc:1034] running on GCE node
I20260620 06:02:50.935030 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:50.935698 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:50.936890 13114 hybrid_clock.cc:648] HybridClock initialized: now 1781935370936877 us; error 40 us; skew 500 ppm
I20260620 06:02:50.938898 13114 webserver.cc:460] Webserver started at http://127.3.115.3:34273/ using document root <none> and password file <none>
I20260620 06:02:50.939488 13114 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:50.939581 13114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:50.939818 13114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:50.941977 13114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "0cbebb88fb6a4ada9ade0079329b433f"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "a1cd37c5a50ed6fe0c4a1581f35f2765"
server_key_iv: "240079448b838c2dcb44f3e5be2310ae"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.942548 13114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "0cbebb88fb6a4ada9ade0079329b433f"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "a1cd37c5a50ed6fe0c4a1581f35f2765"
server_key_iv: "240079448b838c2dcb44f3e5be2310ae"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.946429 13114 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:02:50.949115 13128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.950654 13114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:50.950806 13114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "0cbebb88fb6a4ada9ade0079329b433f"
format_stamp: "Formatted at 2026-06-20 06:02:50 on dist-test-slave-4h14"
server_key: "a1cd37c5a50ed6fe0c4a1581f35f2765"
server_key_iv: "240079448b838c2dcb44f3e5be2310ae"
server_key_version: "encryptionkey@0"
I20260620 06:02:50.950949 13114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:50.968932 13114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:50.969756 13114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:50.969983 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:50.970674 13114 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:50.971678 13114 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:50.971752 13114 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.971830 13114 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:50.971881 13114 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:50.980909 13114 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:45835
I20260620 06:02:50.980929 13241 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:45835 every 8 connection(s)
I20260620 06:02:50.981870 13114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:50.991310 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.983776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:51697 (local address 127.3.115.62:33663)
0620 06:02:50.983942 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:50.983946 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:50.985082 (+ 1136us) server_negotiation.cc:407] Connection header received
0620 06:02:50.986516 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:50.986520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:50.986580 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:50.986685 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:50.987641 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.988174 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.989367 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.989601 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.989717 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.990799 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:50.990802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:50.990805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:50.990825 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:50.990843 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:50.990851 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:50.990930 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:50.991097 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 06:02:50.991159 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260620 06:02:50.992141 13244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:50.984159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:33663 (local address 127.3.115.3:51697)
0620 06:02:50.984897 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:50.984952 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:50.986272 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:50.986868 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:50.986876 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:50.986972 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:50.987481 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.987494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.988301 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.988305 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.989232 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:50.989241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:50.989757 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:50.989760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:50.989925 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:50.990536 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:50.990537 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:50.990578 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:50.990641 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:50.990931 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:50.990953 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:50.990960 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 06:02:50.991095 (+ 135us) client_negotiation.cc:240] Negotiation successful
0620 06:02:50.991377 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":118,"threads_started":1}
I20260620 06:02:50.992151 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13114
I20260620 06:02:50.992270 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:50.992623 3532 external_mini_cluster.cc:1428] Setting key 8be71def8f24fcd426603fabd9750d4f
I20260620 06:02:50.993460 13242 heartbeater.cc:347] Connected to a master server at 127.3.115.62:33663
I20260620 06:02:50.993707 13242 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:50.994257 13242 heartbeater.cc:511] Master 127.3.115.62:33663 requested a full tablet report, sending...
I20260620 06:02:50.995317 12792 ts_manager.cc:194] Registered new tserver with Master: 0cbebb88fb6a4ada9ade0079329b433f (127.3.115.3:45835)
I20260620 06:02:50.995910 12792 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.3:51697
I20260620 06:02:51.006597 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:02:51.009661 12831 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:51.008628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57410 (local address 127.3.115.62:33663)
0620 06:02:51.008802 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:51.008806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:51.009022 (+ 216us) server_negotiation.cc:407] Connection header received
0620 06:02:51.009137 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:51.009139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:51.009183 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:51.009250 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:51.009525 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57410: BlockingRecv error: recv got EOF from 127.0.0.1:57410 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260620 06:02:51.009776 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12852
I20260620 06:02:51.015547 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12983
I20260620 06:02:51.021062 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13114
I20260620 06:02:51.026576 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 12761
2026-06-20T06:02:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (902 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T06:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:51Z Disabled control of system clock
I20260620 06:02:51.055649 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42117
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:02:51.166177 13253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:51.166435 13253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:51.166472 13253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:51.170485 13253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:51.170540 13253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:51.170555 13253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:51.170572 13253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:51.170588 13253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:51.175274 13253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:51.176321 13253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:51.177698 13253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:51.186226 13261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.186218 13258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.186214 13259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:51.186439 13253 server_base.cc:1034] running on GCE node
I20260620 06:02:51.187146 13253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:51.188278 13253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:51.189436 13253 hybrid_clock.cc:648] HybridClock initialized: now 1781935371189390 us; error 61 us; skew 500 ppm
I20260620 06:02:51.191511 13253 webserver.cc:460] Webserver started at http://127.3.115.62:44205/ using document root <none> and password file <none>
I20260620 06:02:51.192150 13253 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:51.192219 13253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:51.192469 13253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:51.194711 13253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "5f03010c400949c6a5c1298aba4ac32b"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "8784ec71767b05f701ab9f325762e7f8"
server_key_iv: "3e6a5fa3b84a36dc2b763090eea495ac"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.195307 13253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "5f03010c400949c6a5c1298aba4ac32b"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "8784ec71767b05f701ab9f325762e7f8"
server_key_iv: "3e6a5fa3b84a36dc2b763090eea495ac"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.199234 13253 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 06:02:51.201908 13267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.203322 13253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 06:02:51.203552 13253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "5f03010c400949c6a5c1298aba4ac32b"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "8784ec71767b05f701ab9f325762e7f8"
server_key_iv: "3e6a5fa3b84a36dc2b763090eea495ac"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.203684 13253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:51.208495 13253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:51.209175 13253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:51.209363 13253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:51.216336 13253 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42117
I20260620 06:02:51.216343 13319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42117 every 8 connection(s)
I20260620 06:02:51.217437 13253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:51.220628 13320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:51.221264 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13253
I20260620 06:02:51.221411 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:51.221725 3532 external_mini_cluster.cc:1428] Setting key adaec65b5c512fdd2b81b5187d48cdd2
I20260620 06:02:51.226958 13320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b: Bootstrap starting.
I20260620 06:02:51.227454 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.223119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.3.115.62:42117)
0620 06:02:51.223716 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:51.223731 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:51.223766 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:02:51.224428 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:51.224445 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:51.224871 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:51.225162 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:51.226001 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:51.226006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:51.226018 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:51.226028 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:51.226081 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:51.226096 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:51.226207 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:51.226323 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 06:02:51.226667 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":124,"threads_started":1}
I20260620 06:02:51.229454 13320 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:51.230295 13320 log.cc:824] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:51.232496 13320 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b: No bootstrap required, opened a new log
I20260620 06:02:51.236068 13320 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f03010c400949c6a5c1298aba4ac32b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42117 } }
I20260620 06:02:51.236323 13320 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:51.236411 13320 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f03010c400949c6a5c1298aba4ac32b, State: Initialized, Role: FOLLOWER
I20260620 06:02:51.236943 13320 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f03010c400949c6a5c1298aba4ac32b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42117 } }
I20260620 06:02:51.237102 13320 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:51.237172 13320 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:51.237272 13320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:51.238368 13320 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f03010c400949c6a5c1298aba4ac32b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42117 } }
I20260620 06:02:51.238716 13320 leader_election.cc:302] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f03010c400949c6a5c1298aba4ac32b; no voters:
I20260620 06:02:51.239118 13320 leader_election.cc:288] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:51.239274 13325 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:51.239562 13325 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [term 1 LEADER]: Becoming Leader. State: Replica: 5f03010c400949c6a5c1298aba4ac32b, State: Running, Role: LEADER
I20260620 06:02:51.239862 13325 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f03010c400949c6a5c1298aba4ac32b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42117 } }
I20260620 06:02:51.240363 13320 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:51.241657 13326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f03010c400949c6a5c1298aba4ac32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f03010c400949c6a5c1298aba4ac32b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42117 } } }
I20260620 06:02:51.241659 13327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f03010c400949c6a5c1298aba4ac32b. Latest consensus state: current_term: 1 leader_uuid: "5f03010c400949c6a5c1298aba4ac32b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f03010c400949c6a5c1298aba4ac32b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42117 } } }
I20260620 06:02:51.241832 13327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:51.242084 13326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:51.242261 13334 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:51.245064 13334 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:51.251076 13334 catalog_manager.cc:1347] Generated new cluster ID: 6f8316d9794842c69d815bc94d6c0c52
I20260620 06:02:51.251163 13334 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:51.259584 13334 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:51.259935 13334 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:51.266202 13334 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f03010c400949c6a5c1298aba4ac32b: Generated new TSK 0
I20260620 06:02:51.266985 13334 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:02:51.277343 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42117
--builtin_ntp_servers=127.3.115.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:02:51.408380 13344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:51.408706 13344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:51.408751 13344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:51.413012 13344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:51.413074 13344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:51.413132 13344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:51.418042 13344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:51.419127 13344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:51.420481 13344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:51.430680 13350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.430732 13349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.430737 13352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:51.430835 13344 server_base.cc:1034] running on GCE node
I20260620 06:02:51.431494 13344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:51.432170 13344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:51.433355 13344 hybrid_clock.cc:648] HybridClock initialized: now 1781935371433319 us; error 50 us; skew 500 ppm
I20260620 06:02:51.435391 13344 webserver.cc:460] Webserver started at http://127.3.115.1:32855/ using document root <none> and password file <none>
I20260620 06:02:51.436167 13344 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:51.436266 13344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:51.436519 13344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:51.438818 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "dcbdc2d898524b14992fb90c35a2be4c"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "7850bcd309f7a6b1a2e5c82433c4e19d"
server_key_iv: "5684f04f5a1900bde8bf4b56aeb5a43c"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.439440 13344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "dcbdc2d898524b14992fb90c35a2be4c"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "7850bcd309f7a6b1a2e5c82433c4e19d"
server_key_iv: "5684f04f5a1900bde8bf4b56aeb5a43c"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.443876 13344 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:02:51.446779 13358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.448175 13344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:51.448316 13344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "dcbdc2d898524b14992fb90c35a2be4c"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "7850bcd309f7a6b1a2e5c82433c4e19d"
server_key_iv: "5684f04f5a1900bde8bf4b56aeb5a43c"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.448441 13344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:51.453217 13344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:51.453826 13344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:51.454020 13344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:51.454576 13344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:51.455495 13344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:51.455574 13344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.455644 13344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:51.455695 13344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.465200 13344 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:41513
I20260620 06:02:51.465256 13471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:41513 every 8 connection(s)
I20260620 06:02:51.466274 13344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:51.473126 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.468427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:46375 (local address 127.3.115.62:42117)
0620 06:02:51.468576 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:51.468580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:51.469861 (+ 1281us) server_negotiation.cc:407] Connection header received
0620 06:02:51.471223 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:51.471226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:51.471267 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:51.471364 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:51.472551 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:51.472554 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:51.472556 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:51.472569 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:51.472618 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:51.472628 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:51.472811 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:51.472956 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 06:02:51.473004 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:02:51.474057 13474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.468950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42117 (local address 127.3.115.1:46375)
0620 06:02:51.469673 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:51.469717 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:51.470962 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:51.471555 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:51.471563 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:51.471664 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:51.472272 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:51.472274 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:51.472319 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:51.472395 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:51.472799 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:51.472807 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:51.472812 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 06:02:51.472961 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:02:51.473185 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":130,"threads_started":1}
I20260620 06:02:51.474704 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13344
I20260620 06:02:51.474850 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:51.475232 3532 external_mini_cluster.cc:1428] Setting key 527a96f923dd8c9b88cfe20e19eecbb7
I20260620 06:02:51.475490 13472 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42117
I20260620 06:02:51.475742 13472 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:51.476202 13472 heartbeater.cc:511] Master 127.3.115.62:42117 requested a full tablet report, sending...
I20260620 06:02:51.477339 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42117
--builtin_ntp_servers=127.3.115.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260620 06:02:51.477905 13284 ts_manager.cc:194] Registered new tserver with Master: dcbdc2d898524b14992fb90c35a2be4c (127.3.115.1:41513)
W20260620 06:02:51.596588 13475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:51.596952 13475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:51.596993 13475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:51.601052 13475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:51.601118 13475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:51.601178 13475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:51.606055 13475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:51.607151 13475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:51.609148 13475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:51.619489 13483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.619483 13480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:51.619596 13475 server_base.cc:1034] running on GCE node
W20260620 06:02:51.619484 13481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:51.620277 13475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:51.621074 13475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:51.622287 13475 hybrid_clock.cc:648] HybridClock initialized: now 1781935371622254 us; error 51 us; skew 500 ppm
I20260620 06:02:51.624368 13475 webserver.cc:460] Webserver started at http://127.3.115.2:36431/ using document root <none> and password file <none>
I20260620 06:02:51.625083 13475 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:51.625185 13475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:51.625425 13475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:51.627874 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "04fd148a06794555bc9e7c4c340dcd68"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "8eefc19570d59448b92ab07ea42e5012"
server_key_iv: "963fb38df85b2f726551bd254765fa3b"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.628479 13475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "04fd148a06794555bc9e7c4c340dcd68"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "8eefc19570d59448b92ab07ea42e5012"
server_key_iv: "963fb38df85b2f726551bd254765fa3b"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.632499 13475 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:02:51.635383 13489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.636960 13475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:02:51.637106 13475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "04fd148a06794555bc9e7c4c340dcd68"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "8eefc19570d59448b92ab07ea42e5012"
server_key_iv: "963fb38df85b2f726551bd254765fa3b"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.637234 13475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:51.642249 13475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:51.642921 13475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:51.643121 13475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:51.643692 13475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:51.644742 13475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:51.644804 13475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.644874 13475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:51.644915 13475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.654181 13475 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:41753
I20260620 06:02:51.654207 13602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:41753 every 8 connection(s)
I20260620 06:02:51.655144 13475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:51.661612 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.657149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:46743 (local address 127.3.115.62:42117)
0620 06:02:51.657295 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:51.657298 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:51.658376 (+ 1078us) server_negotiation.cc:407] Connection header received
0620 06:02:51.659748 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:51.659752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:51.659799 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:51.659910 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:51.661122 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:51.661126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:51.661129 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:51.661143 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:51.661170 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:51.661179 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:51.661276 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:51.661423 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 06:02:51.661469 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:02:51.662532 13605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.657549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42117 (local address 127.3.115.2:46743)
0620 06:02:51.658212 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:51.658254 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:51.659507 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:51.660085 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:51.660093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:51.660185 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:51.660841 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:51.660843 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:51.660883 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:51.660953 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:51.661267 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:51.661275 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:51.661281 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:02:51.661466 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 06:02:51.661718 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":123,"threads_started":1}
I20260620 06:02:51.663755 13603 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42117
I20260620 06:02:51.664026 13603 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:51.664477 13603 heartbeater.cc:511] Master 127.3.115.62:42117 requested a full tablet report, sending...
I20260620 06:02:51.664898 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13475
I20260620 06:02:51.665009 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:51.665313 3532 external_mini_cluster.cc:1428] Setting key a4c5ebbf5affbe6293009a548e047a38
I20260620 06:02:51.665748 13284 ts_manager.cc:194] Registered new tserver with Master: 04fd148a06794555bc9e7c4c340dcd68 (127.3.115.2:41753)
I20260620 06:02:51.667689 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42117
--builtin_ntp_servers=127.3.115.20:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 06:02:51.781834 13606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:51.782147 13606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:51.782188 13606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:51.786398 13606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:51.786458 13606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:51.786518 13606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:51.791417 13606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13606
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:51.792560 13606 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:51.794004 13606 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:51.804457 13614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.804441 13612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:51.804441 13611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:51.804737 13606 server_base.cc:1034] running on GCE node
I20260620 06:02:51.805245 13606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:51.805908 13606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:51.807104 13606 hybrid_clock.cc:648] HybridClock initialized: now 1781935371807087 us; error 30 us; skew 500 ppm
I20260620 06:02:51.809172 13606 webserver.cc:460] Webserver started at http://127.3.115.3:36883/ using document root <none> and password file <none>
I20260620 06:02:51.809779 13606 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:51.809846 13606 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:51.810091 13606 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:51.812271 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "54d81ad806c34cbdad817109e04cc266"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "2d0b1583827779118d4e81fc1ee1ee93"
server_key_iv: "accb7b6e32e547fcbb995589d1e05a45"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.812889 13606 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "54d81ad806c34cbdad817109e04cc266"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "2d0b1583827779118d4e81fc1ee1ee93"
server_key_iv: "accb7b6e32e547fcbb995589d1e05a45"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.816819 13606 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:51.819361 13620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.820573 13606 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:51.820745 13606 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "54d81ad806c34cbdad817109e04cc266"
format_stamp: "Formatted at 2026-06-20 06:02:51 on dist-test-slave-4h14"
server_key: "2d0b1583827779118d4e81fc1ee1ee93"
server_key_iv: "accb7b6e32e547fcbb995589d1e05a45"
server_key_version: "encryptionkey@0"
I20260620 06:02:51.820890 13606 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:51.825519 13606 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:51.826164 13606 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:51.826341 13606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:51.826884 13606 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:51.827785 13606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:51.827855 13606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.827924 13606 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:51.827976 13606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:51.837159 13606 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:44711
I20260620 06:02:51.837210 13733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:44711 every 8 connection(s)
I20260620 06:02:51.838145 13606 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:51.844100 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13606
I20260620 06:02:51.844203 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:51.844508 3532 external_mini_cluster.cc:1428] Setting key 07213fa9a85d533ba764abd634cbc4b9
I20260620 06:02:51.844952 13323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.840092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:35089 (local address 127.3.115.62:42117)
0620 06:02:51.840263 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:51.840268 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:51.841651 (+ 1383us) server_negotiation.cc:407] Connection header received
0620 06:02:51.843091 (+ 1440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:51.843095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:51.843135 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:51.843218 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:51.844418 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:51.844422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:51.844425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:02:51.844437 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:51.844460 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:02:51.844476 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:51.844583 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:51.844766 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 06:02:51.844815 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:02:51.846009 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:51.840552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42117 (local address 127.3.115.3:35089)
0620 06:02:51.841462 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:51.841507 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:51.842869 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:51.843358 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:51.843365 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:51.843459 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:51.844072 (+ 613us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:02:51.844073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:51.844118 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:02:51.844231 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:51.844568 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:51.844578 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:51.844584 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:02:51.844814 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 06:02:51.845094 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":215,"threads_started":1}
I20260620 06:02:51.847172 13734 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42117
I20260620 06:02:51.847419 13734 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:51.847867 13734 heartbeater.cc:511] Master 127.3.115.62:42117 requested a full tablet report, sending...
I20260620 06:02:51.848922 13283 ts_manager.cc:194] Registered new tserver with Master: 54d81ad806c34cbdad817109e04cc266 (127.3.115.3:44711)
I20260620 06:02:51.858538 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 06:02:51.861310 13323 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:02:51.860417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55398 (local address 127.3.115.62:42117)
0620 06:02:51.860524 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:51.860528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:51.860773 (+ 245us) server_negotiation.cc:407] Connection header received
0620 06:02:51.860890 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:51.860892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:51.860926 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:51.860994 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:51.861205 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55398: BlockingRecv error: recv got EOF from 127.0.0.1:55398 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:02:51.861630 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13344
I20260620 06:02:51.867658 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13475
I20260620 06:02:51.872891 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13606
I20260620 06:02:51.878262 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13253
2026-06-20T06:02:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (851 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:51 dist-test-slave-4h14 krb5kdc[13744](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:51 dist-test-slave-4h14 krb5kdc[13744](info): set up 2 sockets
Jun 20 06:02:51 dist-test-slave-4h14 krb5kdc[13744](info): commencing operation
krb5kdc: starting...
W20260620 06:02:52.930279 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:02:53.057065 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:39599
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:39599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:53.169595 13757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:53.169902 13757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:53.169984 13757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:53.174034 13757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:02:53.174122 13757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:53.174160 13757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:53.174218 13757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:02:53.174259 13757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:02:53.179551 13757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:39599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:39599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:53.180771 13757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:53.181882 13757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:02:53.190508 13757 server_base.cc:1034] running on GCE node
W20260620 06:02:53.190454 13765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.190459 13763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.190464 13762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:53.191390 13757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:53.192619 13757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:53.193857 13757 hybrid_clock.cc:648] HybridClock initialized: now 1781935373193822 us; error 52 us; skew 500 ppm
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:53.196849 13757 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:02:53.197749 13757 webserver.cc:460] Webserver started at http://127.3.115.62:37689/ using document root <none> and password file <none>
I20260620 06:02:53.198324 13757 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:53.198417 13757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:53.198657 13757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:53.200821 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "54c38f377bb648c1a161cd208989e813"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "2acdde1bd254b6892d697fde0533f1d7"
server_key_iv: "58ee5d4243feacdc46dc133495a80202"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.201380 13757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "54c38f377bb648c1a161cd208989e813"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "2acdde1bd254b6892d697fde0533f1d7"
server_key_iv: "58ee5d4243feacdc46dc133495a80202"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.205281 13757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260620 06:02:53.207808 13772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.209173 13757 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:53.209322 13757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "54c38f377bb648c1a161cd208989e813"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "2acdde1bd254b6892d697fde0533f1d7"
server_key_iv: "58ee5d4243feacdc46dc133495a80202"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.209439 13757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:53.222462 13757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:53.225911 13757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:53.226123 13757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:53.233273 13757 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:39599
I20260620 06:02:53.233630 13824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:39599 every 8 connection(s)
I20260620 06:02:53.234324 13757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:02:53.237232 13825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:53.242908 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813: Bootstrap starting.
I20260620 06:02:53.243780 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13757
I20260620 06:02:53.243916 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:02:53.244187 3532 external_mini_cluster.cc:1428] Setting key 00e7f431f87e9ca3074355f42f19dbfd
I20260620 06:02:53.245847 13825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:53.246627 13825 log.cc:824] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:53.249092 13825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813: No bootstrap required, opened a new log
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:53.253855 13825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c38f377bb648c1a161cd208989e813" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39599 } }
I20260620 06:02:53.254200 13825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:53.254295 13825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54c38f377bb648c1a161cd208989e813, State: Initialized, Role: FOLLOWER
I20260620 06:02:53.254825 13825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c38f377bb648c1a161cd208989e813" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39599 } }
I20260620 06:02:53.254998 13825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:02:53.255074 13825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:02:53.255178 13825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:53.256126 13825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c38f377bb648c1a161cd208989e813" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39599 } }
I20260620 06:02:53.256507 13825 leader_election.cc:302] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54c38f377bb648c1a161cd208989e813; no voters:
I20260620 06:02:53.256901 13825 leader_election.cc:288] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:02:53.256963 13830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:53.257187 13830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [term 1 LEADER]: Becoming Leader. State: Replica: 54c38f377bb648c1a161cd208989e813, State: Running, Role: LEADER
I20260620 06:02:53.257582 13830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c38f377bb648c1a161cd208989e813" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39599 } }
I20260620 06:02:53.258317 13825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:02:53.259479 13832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54c38f377bb648c1a161cd208989e813. Latest consensus state: current_term: 1 leader_uuid: "54c38f377bb648c1a161cd208989e813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c38f377bb648c1a161cd208989e813" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39599 } } }
I20260620 06:02:53.259617 13832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:53.259749 13831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54c38f377bb648c1a161cd208989e813" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54c38f377bb648c1a161cd208989e813" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 39599 } } }
I20260620 06:02:53.259919 13831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813 [sys.catalog]: This master's current role is: LEADER
I20260620 06:02:53.260078 13839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:02:53.261159 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:53.245423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49752 (local address 127.3.115.62:39599)
0620 06:02:53.245884 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:53.245900 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:53.245937 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:02:53.246639 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:53.246658 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:53.247101 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:53.247516 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:53.248504 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.249616 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.250825 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.251353 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.251510 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:53.254665 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:53.254670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:53.254686 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:53.254718 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:53.256769 (+ 2051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:53.257360 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:53.257366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:53.257372 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:53.257551 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:53.257871 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:53.257874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:53.257876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:53.258294 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:53.258536 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:53.258854 (+ 318us) server_negotiation.cc:299] Negotiation successful
0620 06:02:53.259195 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":149,"threads_started":1}
I20260620 06:02:53.263221 13839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:02:53.268728 13839 catalog_manager.cc:1347] Generated new cluster ID: c1bb290e076e44aab635f141165d15a8
I20260620 06:02:53.268792 13839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:02:53.295011 13839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:02:53.296398 13839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:02:53.303464 13839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813: Generated new TSK 0
I20260620 06:02:53.304368 13839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:02:53.341693 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39599
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:53.456880 13851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:53.457257 13851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:53.457304 13851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:53.461817 13851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:53.461894 13851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:53.461990 13851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:02:53.467685 13851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:53.468907 13851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:53.469971 13851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:53.480031 13857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.480024 13856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.480028 13859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:53.480276 13851 server_base.cc:1034] running on GCE node
I20260620 06:02:53.480830 13851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:53.481524 13851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:53.482728 13851 hybrid_clock.cc:648] HybridClock initialized: now 1781935373482707 us; error 30 us; skew 500 ppm
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:53.485780 13851 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:02:53.486711 13851 webserver.cc:460] Webserver started at http://127.3.115.1:36393/ using document root <none> and password file <none>
I20260620 06:02:53.487239 13851 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:53.487296 13851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:53.487460 13851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:53.489588 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "644bea76d3d34a669a045515aa63fdbc"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "5cff13ab3b61302c1ca2f365624c87b6"
server_key_iv: "e17d4ff3adfe2ccbbf6e35f06b929cb5"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.490154 13851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "644bea76d3d34a669a045515aa63fdbc"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "5cff13ab3b61302c1ca2f365624c87b6"
server_key_iv: "e17d4ff3adfe2ccbbf6e35f06b929cb5"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.493973 13851 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:02:53.496508 13866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.497782 13851 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:02:53.497938 13851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "644bea76d3d34a669a045515aa63fdbc"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "5cff13ab3b61302c1ca2f365624c87b6"
server_key_iv: "e17d4ff3adfe2ccbbf6e35f06b929cb5"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.498051 13851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:53.527998 13851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:53.531677 13851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:53.531842 13851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:53.532407 13851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:53.533244 13851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:53.533294 13851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.533334 13851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:53.533350 13851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.542361 13851 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:43765
I20260620 06:02:53.542397 13979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:43765 every 8 connection(s)
I20260620 06:02:53.543298 13851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:02:53.548813 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13851
I20260620 06:02:53.548940 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:02:53.549280 3532 external_mini_cluster.cc:1428] Setting key 76d53981114b1a063688d94f4866ad9c
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:53.559150 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:53.545214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:60053 (local address 127.3.115.62:39599)
0620 06:02:53.545380 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:53.545384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:53.546281 (+ 897us) server_negotiation.cc:407] Connection header received
0620 06:02:53.547651 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:53.547654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:53.547699 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:53.547781 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:53.549473 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.550125 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.551671 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.552106 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.552225 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:53.554887 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:53.554894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:53.554900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:53.554935 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:53.556749 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:53.557280 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:53.557285 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:53.557288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:53.557384 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:53.557814 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:53.557819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:53.557823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:53.558092 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:53.558209 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:53.558788 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 06:02:53.558927 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:02:53.560230 13982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:53.545629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:39599 (local address 127.3.115.1:60053)
0620 06:02:53.546110 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:53.546157 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:53.547426 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:53.547985 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:53.547994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:53.548383 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:53.549212 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:53.549237 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.550276 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.550279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:53.551515 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:53.551525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.552270 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.552273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:53.552428 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:53.553196 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:53.553213 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:53.554704 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:53.556885 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:53.556890 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:53.556903 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:53.557175 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:53.557550 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:53.557553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:53.557554 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:53.557702 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:53.558216 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:53.558221 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:53.558517 (+ 296us) client_negotiation.cc:769] Sending connection context
0620 06:02:53.558696 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 06:02:53.559224 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260620 06:02:53.561569 13980 heartbeater.cc:347] Connected to a master server at 127.3.115.62:39599
I20260620 06:02:53.561866 13980 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:02:53.562513 13980 heartbeater.cc:511] Master 127.3.115.62:39599 requested a full tablet report, sending...
I20260620 06:02:53.564282 13789 ts_manager.cc:194] Registered new tserver with Master: 644bea76d3d34a669a045515aa63fdbc (127.3.115.1:43765)
I20260620 06:02:53.565506 13789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:60053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:02:53.579186 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39599
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:53.692001 13985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:53.692328 13985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:53.692410 13985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:53.696539 13985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:53.696645 13985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:53.696765 13985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:02:53.702173 13985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:53.703267 13985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:53.704309 13985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:53.713990 13990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.714003 13993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.713997 13991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:53.714311 13985 server_base.cc:1034] running on GCE node
I20260620 06:02:53.714748 13985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:53.715469 13985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:53.716733 13985 hybrid_clock.cc:648] HybridClock initialized: now 1781935373716706 us; error 46 us; skew 500 ppm
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:53.719631 13985 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:02:53.720521 13985 webserver.cc:460] Webserver started at http://127.3.115.2:36165/ using document root <none> and password file <none>
I20260620 06:02:53.721130 13985 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:53.721191 13985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:53.721401 13985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:53.723419 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "218ea2d842c04bb4a64c747549d0de23"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "36a4c26c245ffa0793bfbfba73a8ab9d"
server_key_iv: "377eef0b444685ede84c74cc7a46de25"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.723977 13985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "218ea2d842c04bb4a64c747549d0de23"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "36a4c26c245ffa0793bfbfba73a8ab9d"
server_key_iv: "377eef0b444685ede84c74cc7a46de25"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.727667 13985 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:02:53.730412 14000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.731809 13985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:02:53.731945 13985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "218ea2d842c04bb4a64c747549d0de23"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "36a4c26c245ffa0793bfbfba73a8ab9d"
server_key_iv: "377eef0b444685ede84c74cc7a46de25"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.732075 13985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:53.780601 13985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:53.784691 13985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:53.784909 13985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:53.785578 13985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:53.786516 13985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:53.786562 13985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.786630 13985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:53.786664 13985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.795027 13985 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:42839
I20260620 06:02:53.795066 14113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:42839 every 8 connection(s)
I20260620 06:02:53.795915 13985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:02:53.796319 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 13985
I20260620 06:02:53.796433 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:02:53.796793 3532 external_mini_cluster.cc:1428] Setting key 1c8ee8460e75d02db9959590598281b7
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:02:53.811210 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:53.798141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:37117 (local address 127.3.115.62:39599)
0620 06:02:53.798269 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:53.798273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:53.799214 (+ 941us) server_negotiation.cc:407] Connection header received
0620 06:02:53.801059 (+ 1845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:53.801062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:53.801108 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:53.801184 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:53.802532 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.803189 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.804455 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.804739 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.804851 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:53.807479 (+ 2628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:53.807482 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:53.807484 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:53.807509 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:53.808924 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:53.809606 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:53.809612 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:53.809614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:53.809700 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:53.810092 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:53.810102 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:53.810104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:53.810280 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:53.810390 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:53.810898 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 06:02:53.811028 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:02:53.812034 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:53.798594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:39599 (local address 127.3.115.2:37117)
0620 06:02:53.799049 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:53.799101 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:53.800812 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:53.801381 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:53.801388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:53.801781 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:53.802320 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:53.802335 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.803328 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.803331 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:53.804308 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:53.804318 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:53.804888 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:53.804892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:53.805066 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:53.805675 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:53.805691 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:53.807271 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:53.809062 (+ 1791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:53.809068 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:53.809084 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:53.809452 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:53.809795 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:53.809800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:53.809802 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:53.809947 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:53.810387 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:53.810394 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:53.810657 (+ 263us) client_negotiation.cc:769] Sending connection context
0620 06:02:53.810842 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 06:02:53.811170 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":125,"threads_started":1}
I20260620 06:02:53.813320 14114 heartbeater.cc:347] Connected to a master server at 127.3.115.62:39599
I20260620 06:02:53.813609 14114 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:53.814232 14114 heartbeater.cc:511] Master 127.3.115.62:39599 requested a full tablet report, sending...
I20260620 06:02:53.815528 13789 ts_manager.cc:194] Registered new tserver with Master: 218ea2d842c04bb4a64c747549d0de23 (127.3.115.2:42839)
I20260620 06:02:53.816359 13789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:37117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:02:53.827348 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39599
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:02:53.939268 14119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:02:53.939544 14119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:02:53.939589 14119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:02:53.943866 14119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:02:53.943935 14119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:02:53.944043 14119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:02:53.949546 14119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:39599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:02:53.950655 14119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:02:53.951726 14119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:02:53.961472 14124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.961455 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:02:53.961455 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:02:53.961573 14119 server_base.cc:1034] running on GCE node
I20260620 06:02:53.962117 14119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:02:53.962700 14119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:02:53.963912 14119 hybrid_clock.cc:648] HybridClock initialized: now 1781935373963873 us; error 49 us; skew 500 ppm
Jun 20 06:02:53 dist-test-slave-4h14 krb5kdc[13744](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:02:53.971140 14119 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:02:53.972133 14119 webserver.cc:460] Webserver started at http://127.3.115.3:45369/ using document root <none> and password file <none>
I20260620 06:02:53.972790 14119 fs_manager.cc:359] Metadata directory not provided
I20260620 06:02:53.972883 14119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:02:53.973140 14119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:02:53.975232 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "f964279b02de4209adf7fe5749f544d0"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "75ed5e116cfd814d5ac8e5fcacfc425b"
server_key_iv: "f47522389ef496e6707ea26816435a60"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.975814 14119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "f964279b02de4209adf7fe5749f544d0"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "75ed5e116cfd814d5ac8e5fcacfc425b"
server_key_iv: "f47522389ef496e6707ea26816435a60"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.979727 14119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:02:53.982355 14134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:53.983759 14119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:02:53.983909 14119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "f964279b02de4209adf7fe5749f544d0"
format_stamp: "Formatted at 2026-06-20 06:02:53 on dist-test-slave-4h14"
server_key: "75ed5e116cfd814d5ac8e5fcacfc425b"
server_key_iv: "f47522389ef496e6707ea26816435a60"
server_key_version: "encryptionkey@0"
I20260620 06:02:53.984031 14119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:02:54.003971 14119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:02:54.007489 14119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:02:54.007699 14119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:02:54.008303 14119 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:02:54.009327 14119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:02:54.009397 14119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:54.009476 14119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:02:54.009528 14119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:02:54.018889 14119 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:36097
I20260620 06:02:54.018910 14247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:36097 every 8 connection(s)
I20260620 06:02:54.019833 14119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:02:54.023962 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 14119
I20260620 06:02:54.024078 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:02:54.024369 3532 external_mini_cluster.cc:1428] Setting key 5fc7743b46d7ab6770e2cfd686d66871
Jun 20 06:02:54 dist-test-slave-4h14 krb5kdc[13744](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935373, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:02:54.035507 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.021896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:59657 (local address 127.3.115.62:39599)
0620 06:02:54.022109 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.022113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.022960 (+ 847us) server_negotiation.cc:407] Connection header received
0620 06:02:54.024467 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.024471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.024532 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.024605 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:54.026234 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.027220 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.028426 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.028716 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.028819 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.031617 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:54.031628 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:54.031631 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:54.031701 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:54.033353 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:54.033925 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:54.033928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:54.033929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:54.034024 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:54.034401 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:54.034403 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:54.034404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:54.034592 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:54.034669 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.035215 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.035371 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260620 06:02:54.036481 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.022316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:39599 (local address 127.3.115.3:59657)
0620 06:02:54.022790 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.022837 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:54.024222 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:54.024845 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:54.024853 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:54.025339 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:02:54.026031 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.026047 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.027334 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.027337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.028303 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.028313 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.028879 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.028882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.029043 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.029689 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:02:54.029706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:02:54.031272 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:02:54.033506 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:54.033512 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:54.033527 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:54.033820 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:54.034136 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:02:54.034138 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:02:54.034140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:02:54.034314 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:02:54.034686 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:02:54.034691 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:02:54.034953 (+ 262us) client_negotiation.cc:769] Sending connection context
0620 06:02:54.035129 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 06:02:54.035529 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":110,"threads_started":1}
I20260620 06:02:54.037847 14248 heartbeater.cc:347] Connected to a master server at 127.3.115.62:39599
I20260620 06:02:54.038120 14248 heartbeater.cc:464] Registering TS with master...
I20260620 06:02:54.038765 14248 heartbeater.cc:511] Master 127.3.115.62:39599 requested a full tablet report, sending...
I20260620 06:02:54.039876 13789 ts_manager.cc:194] Registered new tserver with Master: f964279b02de4209adf7fe5749f544d0 (127.3.115.3:36097)
I20260620 06:02:54.040462 13789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:59657
I20260620 06:02:54.051483 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:02:54.061769 13828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.053532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.3.115.62:39599)
0620 06:02:54.053694 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.053699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.053962 (+ 263us) server_negotiation.cc:407] Connection header received
0620 06:02:54.054011 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.054014 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.054055 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.054137 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:02:54.054885 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.055401 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.056327 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.056538 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.056649 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.058235 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:02:54.058243 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:02:54.058248 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:02:54.058281 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:02:54.059906 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:54.060385 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:54.060389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:54.060391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:54.060478 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:02:54.060832 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:02:54.060835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:02:54.060837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:02:54.061083 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:02:54.061228 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.061466 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.061593 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:02:54.064822 13789 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:02:54.067894 13789 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:02:54.082446 14262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.077164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:36097 (local address 127.0.0.1:34804)
0620 06:02:54.077411 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.077428 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:54.077512 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:54.078536 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:54.078539 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:54.078556 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:54.078648 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.078654 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.079620 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.079627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.080876 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.080885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.081851 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.081854 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.081969 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.082013 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 06:02:54.082134 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 06:02:54.082256 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260620 06:02:54.082453 14263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.077647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34804 (local address 127.3.115.3:36097)
0620 06:02:54.078087 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.078098 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.078119 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:02:54.078192 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.078199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.078381 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.078555 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:54.078772 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.079497 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.081002 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.081728 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.081836 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.081982 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.082181 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.082291 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":165,"threads_started":1}
I20260620 06:02:54.082661 14261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.076422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:43765 (local address 127.0.0.1:53528)
0620 06:02:54.076942 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.076993 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:54.077125 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:54.078295 (+ 1170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:54.078301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:54.078325 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:54.078470 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.078479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.079992 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.079996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.081240 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.081250 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.082279 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.082283 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.082368 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.082390 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:02:54.082437 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 06:02:54.082515 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"spinlock_wait_cycles":8192,"thread_start_us":93,"threads_started":1}
I20260620 06:02:54.082870 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.076541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42839 (local address 127.0.0.1:57564)
0620 06:02:54.076942 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.076991 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:54.077126 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:54.078644 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:54.078647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:54.078664 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:54.078768 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.078773 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.080300 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.080304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.081418 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.081431 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.082449 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.082454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.082558 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.082588 (+ 30us) client_negotiation.cc:769] Sending connection context
0620 06:02:54.082646 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 06:02:54.082732 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":2,"spinlock_wait_cycles":2816,"thread_start_us":142,"threads_started":1}
I20260620 06:02:54.083297 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.076543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.3.115.1:43765)
0620 06:02:54.077317 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.077325 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.077350 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:02:54.077841 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.077848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.078094 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.078280 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:54.078629 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.079836 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.081364 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.082145 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.082845 (+ 700us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.083008 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.083071 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.083158 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":83,"threads_started":1}
I20260620 06:02:54.084372 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.077902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.3.115.2:42839)
0620 06:02:54.078189 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.078199 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.078218 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:02:54.078305 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.078310 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.078495 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.078905 (+ 410us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:54.079074 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.080161 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.081527 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.082307 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.083983 (+ 1676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.084086 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.084126 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.084238 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":117,"threads_started":1}
I20260620 06:02:54.086083 13914 tablet_service.cc:1467] Processing CreateTablet for tablet 5148cac7c8f048df8867d9a822255aac (DEFAULT_TABLE table=test-table [id=38853e9befee4aab914f2211b22349ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:54.086109 14182 tablet_service.cc:1467] Processing CreateTablet for tablet 5148cac7c8f048df8867d9a822255aac (DEFAULT_TABLE table=test-table [id=38853e9befee4aab914f2211b22349ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:54.087167 13914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5148cac7c8f048df8867d9a822255aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:54.087186 14182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5148cac7c8f048df8867d9a822255aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:54.086936 14048 tablet_service.cc:1467] Processing CreateTablet for tablet 5148cac7c8f048df8867d9a822255aac (DEFAULT_TABLE table=test-table [id=38853e9befee4aab914f2211b22349ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:02:54.087850 14048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5148cac7c8f048df8867d9a822255aac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:02:54.093437 14265 tablet_bootstrap.cc:492] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc: Bootstrap starting.
I20260620 06:02:54.095045 14267 tablet_bootstrap.cc:492] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Bootstrap starting.
I20260620 06:02:54.095791 14265 tablet_bootstrap.cc:654] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:54.096529 14265 log.cc:824] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:54.096997 14266 tablet_bootstrap.cc:492] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Bootstrap starting.
I20260620 06:02:54.097316 14267 tablet_bootstrap.cc:654] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:54.098115 14267 log.cc:824] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:54.098877 14265 tablet_bootstrap.cc:492] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc: No bootstrap required, opened a new log
I20260620 06:02:54.099097 14265 ts_tablet_manager.cc:1397] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:02:54.099524 14266 tablet_bootstrap.cc:654] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Neither blocks nor log segments found. Creating new log.
I20260620 06:02:54.100298 14266 log.cc:824] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Log is configured to *not* fsync() on all Append() calls
I20260620 06:02:54.102299 14267 tablet_bootstrap.cc:492] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: No bootstrap required, opened a new log
I20260620 06:02:54.102299 14266 tablet_bootstrap.cc:492] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: No bootstrap required, opened a new log
I20260620 06:02:54.102474 14266 ts_tablet_manager.cc:1397] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:02:54.102474 14267 ts_tablet_manager.cc:1397] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260620 06:02:54.103406 14265 raft_consensus.cc:348] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.103611 14265 raft_consensus.cc:374] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:54.103650 14265 raft_consensus.cc:729] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 644bea76d3d34a669a045515aa63fdbc, State: Initialized, Role: FOLLOWER
I20260620 06:02:54.104038 14265 consensus_queue.cc:260] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.104638 13980 heartbeater.cc:503] Master 127.3.115.62:39599 was elected leader, sending a full tablet report...
I20260620 06:02:54.105016 14265 ts_tablet_manager.cc:1428] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:02:54.106619 14266 raft_consensus.cc:348] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.106606 14267 raft_consensus.cc:348] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.106848 14266 raft_consensus.cc:374] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:54.106856 14267 raft_consensus.cc:374] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:02:54.106899 14266 raft_consensus.cc:729] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f964279b02de4209adf7fe5749f544d0, State: Initialized, Role: FOLLOWER
I20260620 06:02:54.106925 14267 raft_consensus.cc:729] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218ea2d842c04bb4a64c747549d0de23, State: Initialized, Role: FOLLOWER
I20260620 06:02:54.107319 14266 consensus_queue.cc:260] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.107398 14267 consensus_queue.cc:260] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.108232 14267 ts_tablet_manager.cc:1428] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:02:54.108302 14266 ts_tablet_manager.cc:1428] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260620 06:02:54.108590 14114 heartbeater.cc:503] Master 127.3.115.62:39599 was elected leader, sending a full tablet report...
I20260620 06:02:54.108721 14248 heartbeater.cc:503] Master 127.3.115.62:39599 was elected leader, sending a full tablet report...
I20260620 06:02:54.191905 14272 raft_consensus.cc:482] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:02:54.192153 14272 raft_consensus.cc:504] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.193425 14272 leader_election.cc:288] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 644bea76d3d34a669a045515aa63fdbc (127.3.115.1:43765), 218ea2d842c04bb4a64c747549d0de23 (127.3.115.2:42839)
I20260620 06:02:54.198623 14250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.193765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:43765 (local address 127.3.115.3:36883)
0620 06:02:54.193912 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.193930 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:54.194049 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:54.194330 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:54.194333 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:54.194354 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:54.194553 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.194562 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.195609 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.195612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.197040 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.197047 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.198024 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.198028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.198229 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.198281 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 06:02:54.198366 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:02:54.198495 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":107}
I20260620 06:02:54.198632 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.193918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:37083 (local address 127.3.115.2:42839)
0620 06:02:54.194094 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.194100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.194493 (+ 393us) server_negotiation.cc:407] Connection header received
0620 06:02:54.194580 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.194582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.194650 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.194733 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:54.194981 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.195673 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.197205 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.197859 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.197980 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.198047 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.198414 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.198531 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260620 06:02:54.198622 14260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.193910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:36883 (local address 127.3.115.1:43765)
0620 06:02:54.194057 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.194063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.194083 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:02:54.194148 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.194154 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.194213 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.194314 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:02:54.194678 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.195488 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.197205 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.197859 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.197987 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.198047 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.198414 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.198496 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:02:54.198678 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.193765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42839 (local address 127.3.115.3:37083)
0620 06:02:54.194387 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.194401 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:02:54.194494 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:02:54.194745 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:02:54.194747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:02:54.194761 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:02:54.194861 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.194865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.195791 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.195794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.197036 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:02:54.197047 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.198024 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.198028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:02:54.198203 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.198277 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 06:02:54.198366 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:02:54.198466 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":299,"thread_start_us":143,"threads_started":1}
I20260620 06:02:54.199331 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5148cac7c8f048df8867d9a822255aac" candidate_uuid: "f964279b02de4209adf7fe5749f544d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218ea2d842c04bb4a64c747549d0de23" is_pre_election: true
I20260620 06:02:54.199337 13934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5148cac7c8f048df8867d9a822255aac" candidate_uuid: "f964279b02de4209adf7fe5749f544d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644bea76d3d34a669a045515aa63fdbc" is_pre_election: true
I20260620 06:02:54.199681 13934 raft_consensus.cc:2454] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f964279b02de4209adf7fe5749f544d0 in term 0.
I20260620 06:02:54.199682 14068 raft_consensus.cc:2454] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f964279b02de4209adf7fe5749f544d0 in term 0.
I20260620 06:02:54.200289 14138 leader_election.cc:302] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 218ea2d842c04bb4a64c747549d0de23, 644bea76d3d34a669a045515aa63fdbc, f964279b02de4209adf7fe5749f544d0; no voters:
I20260620 06:02:54.200626 14272 raft_consensus.cc:2781] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:02:54.200780 14272 raft_consensus.cc:482] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:02:54.200870 14272 raft_consensus.cc:3037] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:54.201972 14272 raft_consensus.cc:504] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.202394 14272 leader_election.cc:288] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [CANDIDATE]: Term 1 election: Requested vote from peers 644bea76d3d34a669a045515aa63fdbc (127.3.115.1:43765), 218ea2d842c04bb4a64c747549d0de23 (127.3.115.2:42839)
I20260620 06:02:54.202798 14068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5148cac7c8f048df8867d9a822255aac" candidate_uuid: "f964279b02de4209adf7fe5749f544d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "218ea2d842c04bb4a64c747549d0de23"
I20260620 06:02:54.202798 13934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5148cac7c8f048df8867d9a822255aac" candidate_uuid: "f964279b02de4209adf7fe5749f544d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "644bea76d3d34a669a045515aa63fdbc"
I20260620 06:02:54.202947 14068 raft_consensus.cc:3037] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:54.202947 13934 raft_consensus.cc:3037] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:02:54.204315 14068 raft_consensus.cc:2454] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f964279b02de4209adf7fe5749f544d0 in term 1.
I20260620 06:02:54.204315 13934 raft_consensus.cc:2454] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f964279b02de4209adf7fe5749f544d0 in term 1.
I20260620 06:02:54.204702 14138 leader_election.cc:302] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 218ea2d842c04bb4a64c747549d0de23, f964279b02de4209adf7fe5749f544d0; no voters:
I20260620 06:02:54.204873 14272 raft_consensus.cc:2781] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:02:54.205125 14272 raft_consensus.cc:686] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 1 LEADER]: Becoming Leader. State: Replica: f964279b02de4209adf7fe5749f544d0, State: Running, Role: LEADER
I20260620 06:02:54.205504 14272 consensus_queue.cc:237] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } }
I20260620 06:02:54.208743 13788 catalog_manager.cc:5495] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 reported cstate change: term changed from 0 to 1, leader changed from <none> to f964279b02de4209adf7fe5749f544d0 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "f964279b02de4209adf7fe5749f544d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f964279b02de4209adf7fe5749f544d0" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 36097 } health_report { overall_health: HEALTHY } } }
I20260620 06:02:54.235003 14263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.231218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34806 (local address 127.3.115.3:36097)
0620 06:02:54.231366 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:02:54.231371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:02:54.231385 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:02:54.231540 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:02:54.231543 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:02:54.231599 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:02:54.231726 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:02:54.232011 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.232525 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.233528 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:02:54.233744 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:02:54.233853 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:02:54.234156 (+ 303us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:02:54.234566 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:02:54.234696 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:02:54.234759 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 06:02:54.234835 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:02:54.241109 14068 raft_consensus.cc:1261] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 1 FOLLOWER]: Refusing update from remote peer f964279b02de4209adf7fe5749f544d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:54.241196 13934 raft_consensus.cc:1261] T 5148cac7c8f048df8867d9a822255aac P 644bea76d3d34a669a045515aa63fdbc [term 1 FOLLOWER]: Refusing update from remote peer f964279b02de4209adf7fe5749f544d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:02:54.241882 14275 consensus_queue.cc:1035] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "218ea2d842c04bb4a64c747549d0de23" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42839 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:54.242082 14272 consensus_queue.cc:1035] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "644bea76d3d34a669a045515aa63fdbc" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:02:54.251750 14280 mvcc.cc:205] Tried to move back new op lower bound from 7298807292884115456 to 7298807292748439552. Current Snapshot: MvccSnapshot[applied={T|T < 7298807292884115456}]
I20260620 06:02:54.261291 13787 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:54.261512 13787 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49758:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:02:54.264082 13787 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54c38f377bb648c1a161cd208989e813: Sending DeleteTablet for 3 replicas of tablet 5148cac7c8f048df8867d9a822255aac
I20260620 06:02:54.265035 13914 tablet_service.cc:1514] Processing DeleteTablet for tablet 5148cac7c8f048df8867d9a822255aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:54 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:53528
I20260620 06:02:54.265034 14182 tablet_service.cc:1514] Processing DeleteTablet for tablet 5148cac7c8f048df8867d9a822255aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:54 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:34804
I20260620 06:02:54.265796 14289 tablet_replica.cc:330] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: stopping tablet replica
I20260620 06:02:54.265756 14048 tablet_service.cc:1514] Processing DeleteTablet for tablet 5148cac7c8f048df8867d9a822255aac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:02:54 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:57564
I20260620 06:02:54.266047 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13851
I20260620 06:02:54.266160 14289 raft_consensus.cc:2229] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:02:54.266251 14290 tablet_replica.cc:330] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: stopping tablet replica
I20260620 06:02:54.266535 14290 raft_consensus.cc:2229] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:02:54.266597 14289 raft_consensus.cc:2258] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:54.266837 14290 raft_consensus.cc:2258] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:02:54.268251 14289 ts_tablet_manager.cc:1905] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:54.268251 14290 ts_tablet_manager.cc:1905] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:02:54.271056 14290 ts_tablet_manager.cc:1918] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:54.271057 14289 ts_tablet_manager.cc:1918] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:02:54.271154 14290 log.cc:1196] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/5148cac7c8f048df8867d9a822255aac
I20260620 06:02:54.271162 14289 log.cc:1196] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/5148cac7c8f048df8867d9a822255aac
I20260620 06:02:54.271483 14289 ts_tablet_manager.cc:1939] T 5148cac7c8f048df8867d9a822255aac P f964279b02de4209adf7fe5749f544d0: Deleting consensus metadata
I20260620 06:02:54.271483 14290 ts_tablet_manager.cc:1939] T 5148cac7c8f048df8867d9a822255aac P 218ea2d842c04bb4a64c747549d0de23: Deleting consensus metadata
I20260620 06:02:54.272773 13776 catalog_manager.cc:4841] TS f964279b02de4209adf7fe5749f544d0 (127.3.115.3:36097): tablet 5148cac7c8f048df8867d9a822255aac (table test-table [id=38853e9befee4aab914f2211b22349ba]) successfully deleted
I20260620 06:02:54.273015 13776 catalog_manager.cc:4841] TS 218ea2d842c04bb4a64c747549d0de23 (127.3.115.2:42839): tablet 5148cac7c8f048df8867d9a822255aac (table test-table [id=38853e9befee4aab914f2211b22349ba]) successfully deleted
W20260620 06:02:54.276943 13773 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:43765 (error 108)
I20260620 06:02:54.277140 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13985
I20260620 06:02:54.278839 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:02:54.278483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:43765 (local address 127.0.0.1:53538)
0620 06:02:54.278677 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 06:02:54.278748 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:43765: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20260620 06:02:54.279130 13773 catalog_manager.cc:4584] TS 644bea76d3d34a669a045515aa63fdbc (127.3.115.1:43765): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5148cac7c8f048df8867d9a822255aac: Network error: Client connection negotiation failed: client connection to 127.3.115.1:43765: connect: Connection refused (error 111)
I20260620 06:02:54.283569 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 14119
I20260620 06:02:54.289359 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 13757
2026-06-20T06:02:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2412 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:02:54 dist-test-slave-4h14 krb5kdc[14294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:02:54 dist-test-slave-4h14 krb5kdc[14294](info): set up 2 sockets
Jun 20 06:02:54 dist-test-slave-4h14 krb5kdc[14294](info): commencing operation
krb5kdc: starting...
W20260620 06:02:55.345904 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:02:55 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:02:55Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/rangeradmin_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/rangeradmin_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/rangerlookup_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/rangerlookup_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
I20260620 06:02:55.519724 3532 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVS9eKZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/postgres -l logfile start
2026-06-20 06:02:57.988 UTC [14324] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:02:57.988 UTC [14324] LOG: listening on IPv4 address "127.3.115.20", port 54035
2026-06-20 06:02:57.992 UTC [14324] LOG: listening on Unix socket "/tmp/.s.PGSQL.54035"
2026-06-20 06:02:58.014 UTC [14327] LOG: database system was shut down at 2026-06-20 06:02:56 UTC
2026-06-20 06:02:58.018 UTC [14324] LOG: database system is ready to accept connections
I20260620 06:02:58.985559 3532 mini_postgres.cc:96] Postgres bound to 54035
2026-06-20 06:02:58.992 UTC [14335] FATAL: database "slave" does not exist
127.3.115.20:54035 - accepting connections
I20260620 06:02:58.992745 3532 mini_ranger.cc:161] Starting Ranger...
I20260620 06:02:59.007717 3532 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:02:59.487521 3532 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:02:59.487599 3532 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-admin
2026-06-20 06:02:59,766 [I] DB FLAVOR :POSTGRES
2026-06-20 06:02:59,767 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:02:59,767 [I] Checking connection..
2026-06-20 06:02:59,767 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:03:00,034 [I] Checking connection passed.
2026-06-20 06:03:00,034 [I] --------- Verifying version history table ---------
2026-06-20 06:03:00,034 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:03:00,268 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:03:00,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:03:00,504 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:03:00,505 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:03:00,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:03:00.740 UTC [14399] WARNING: there is no transaction in progress
2026-06-20 06:03:00,746 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:03:00,747 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:03:01,005 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:03:01,006 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:03:01,006 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:03:01,256 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:03:01,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4h14.c.gcp-upstream.internal','N') ;"
2026-06-20 06:03:01,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:03:02,016 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:03:02,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:03:02,289 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:03:02,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:03:02,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:03:02,774 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:03:02,774 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:03:03.775 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.793 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.811 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.830 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.847 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.865 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.932 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.939 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.954 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.965 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.979 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:03.994 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:04.005 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:04.012 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:04.019 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:04.026 UTC [14507] WARNING: there is no transaction in progress
2026-06-20 06:03:04,725 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:03:04,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:03:04,978 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:03:04,978 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:03:05,251 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:03:05,252 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:03:05,502 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:03:05,502 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:03:05,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:03:05,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4h14.c.gcp-upstream.internal';"
2026-06-20 06:03:06,212 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:03:06,213 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:03:06,477 [I] DB_PATCHES have already been applied
I20260620 06:03:06.482839 3532 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVS9eKZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:03:06.482906 3532 mini_ranger.cc:193] Using host: 127.3.115.20
I20260620 06:03:06.486408 3532 mini_ranger.cc:227] Ranger admin URL: http://127.3.115.20:57217
Jun 20, 2026 6:03:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:03:06 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:03:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:03:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:03:06 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:03:07 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:03:07.636951 3532 mini_ranger.cc:160] Time spent starting Ranger: real 8.644s user 0.003s sys 0.008s
Jun 20, 2026 6:03:28 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:03:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:03:28 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:03:28 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:03:29 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:03:30.294777 3532 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/rangerkms_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/rangerkms_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.115.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.115.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:03:30 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935410, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:03:30.385676 3532 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:03:30.396492 3532 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:03:31.471349 3532 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:03:31.471449 3532 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms
I20260620 06:03:31.471541 3532 mini_ranger_kms.cc:227] Using postgres at 127.3.115.20:54035
2026-06-20 06:03:31,664 [I] DB FLAVOR :POSTGRES
2026-06-20 06:03:31,664 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:03:31,664 [I] Checking connection
2026-06-20 06:03:31,664 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:03:31,916 [I] connection success
2026-06-20 06:03:31,916 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:03:31,917 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:03:31,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:03:32,169 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:03:32,170 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:03:32,170 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:03:32,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:54035/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:03:32,436 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:03:32.899147 3532 mini_ranger_kms.cc:310] Created kms service
I20260620 06:03:33.097776 3532 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:03:33.145855 3532 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:03:33 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935410, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:03:33.585345 3532 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:03:33.585510 3532 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:03:33.585538 3532 mini_ranger_kms.cc:240] Using host: 127.3.115.20
I20260620 06:03:33.589753 3532 mini_ranger_kms.cc:276] Ranger KMS PID: 14688
I20260620 06:03:33.589907 3532 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.115.20:38181
Jun 20, 2026 6:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:03:34 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:03:35 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 06:03:35.885445 3532 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.500s user 0.002s sys 0.174s
I20260620 06:03:35.885561 3532 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:03:35.885694 3532 mini_ranger_kms.cc:401] 127.3.115.20:38181/kms/v1/keys
Jun 20 06:03:35 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.115.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 06:03:36 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.115.20@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
Jun 20 06:03:36 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.115.20@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
Jun 20 06:03:39 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:03:39.105877 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34691
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:34691
--ranger_config_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:38181/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:03:39.233170 14738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:03:39.233474 14738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:03:39.233515 14738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:03:39.237614 14738 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:03:39.237689 14738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:03:39.237708 14738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:03:39.237725 14738 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:03:39.237739 14738 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:03:39.243386 14738 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.115.20:38181/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:34691
--ranger_config_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:03:39.244479 14738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:03:39.245672 14738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:03:39.255227 14746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:39.255203 14743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:39.255240 14744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:03:39.255679 14738 server_base.cc:1034] running on GCE node
I20260620 06:03:39.256493 14738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:03:39.257725 14738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:03:39.258893 14738 hybrid_clock.cc:648] HybridClock initialized: now 1781935419258877 us; error 31 us; skew 500 ppm
Jun 20 06:03:39 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935419, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:03:39.261930 14738 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:03:39.263015 14738 webserver.cc:460] Webserver started at http://127.3.115.62:41823/ using document root <none> and password file <none>
I20260620 06:03:39.263691 14738 fs_manager.cc:359] Metadata directory not provided
I20260620 06:03:39.263772 14738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:03:39.263988 14738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:03:39 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935419, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:03:39.329421 14738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "366deb5856e944dcb2ca9697c103d834"
format_stamp: "Formatted at 2026-06-20 06:03:39 on dist-test-slave-4h14"
server_key: "4c2b577ca5f8fe80ddac34a8de353b38"
server_key_iv: "7ae582f73c5330d8bc0ad3264be2fc35"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:39.330078 14738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "366deb5856e944dcb2ca9697c103d834"
format_stamp: "Formatted at 2026-06-20 06:03:39 on dist-test-slave-4h14"
server_key: "4c2b577ca5f8fe80ddac34a8de353b38"
server_key_iv: "7ae582f73c5330d8bc0ad3264be2fc35"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:39.334194 14738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:03:39.352262 14753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:39.354015 14738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:03:39.354168 14738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "366deb5856e944dcb2ca9697c103d834"
format_stamp: "Formatted at 2026-06-20 06:03:39 on dist-test-slave-4h14"
server_key: "4c2b577ca5f8fe80ddac34a8de353b38"
server_key_iv: "7ae582f73c5330d8bc0ad3264be2fc35"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:39.354317 14738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:03:39.377020 14738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:03:39.380723 14738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:03:39.380935 14738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:03:39.390795 14738 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:34691
I20260620 06:03:39.390825 14805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:34691 every 8 connection(s)
I20260620 06:03:39.392125 14738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:03:39.394556 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 14738
I20260620 06:03:39.394697 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:03:39.395031 3532 external_mini_cluster.cc:1428] Setting key 66017d568fd2d4aaf7861e82f41f1112
I20260620 06:03:39.395467 14806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:03:39.402752 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834: Bootstrap starting.
I20260620 06:03:39.405720 14806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834: Neither blocks nor log segments found. Creating new log.
Jun 20 06:03:39 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935419, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:03:39.406579 14806 log.cc:824] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834: Log is configured to *not* fsync() on all Append() calls
I20260620 06:03:39.408836 14806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834: No bootstrap required, opened a new log
I20260620 06:03:39.413111 14809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:39.396516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.3.115.62:34691)
0620 06:03:39.397043 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:39.397064 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:39.397112 (+ 48us) server_negotiation.cc:407] Connection header received
0620 06:03:39.398046 (+ 934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:39.398073 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:39.398537 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:39.398943 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:03:39.399887 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:39.401539 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:39.402758 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:39.403501 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:39.403684 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:39.407001 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:03:39.407009 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:03:39.407030 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:03:39.407070 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:03:39.409687 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:39.410214 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:39.410219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:39.410223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:39.410341 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:39.411014 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:39.411017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:39.411018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:39.411473 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:03:39.411621 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:39.411936 (+ 315us) server_negotiation.cc:299] Negotiation successful
0620 06:03:39.412236 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":180,"threads_started":1}
I20260620 06:03:39.413096 14806 raft_consensus.cc:348] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366deb5856e944dcb2ca9697c103d834" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34691 } }
I20260620 06:03:39.413439 14806 raft_consensus.cc:374] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:03:39.413519 14806 raft_consensus.cc:729] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 366deb5856e944dcb2ca9697c103d834, State: Initialized, Role: FOLLOWER
I20260620 06:03:39.414018 14806 consensus_queue.cc:260] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366deb5856e944dcb2ca9697c103d834" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34691 } }
I20260620 06:03:39.414167 14806 raft_consensus.cc:388] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:03:39.414227 14806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:03:39.414325 14806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:03:39.415263 14806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366deb5856e944dcb2ca9697c103d834" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34691 } }
I20260620 06:03:39.415613 14806 leader_election.cc:302] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 366deb5856e944dcb2ca9697c103d834; no voters:
I20260620 06:03:39.415944 14806 leader_election.cc:288] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:03:39.416064 14811 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:03:39.416309 14811 raft_consensus.cc:686] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [term 1 LEADER]: Becoming Leader. State: Replica: 366deb5856e944dcb2ca9697c103d834, State: Running, Role: LEADER
I20260620 06:03:39.416699 14811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366deb5856e944dcb2ca9697c103d834" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34691 } }
I20260620 06:03:39.417333 14806 sys_catalog.cc:564] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:03:39.418570 14813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 366deb5856e944dcb2ca9697c103d834. Latest consensus state: current_term: 1 leader_uuid: "366deb5856e944dcb2ca9697c103d834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366deb5856e944dcb2ca9697c103d834" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34691 } } }
I20260620 06:03:39.418591 14812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "366deb5856e944dcb2ca9697c103d834" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "366deb5856e944dcb2ca9697c103d834" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34691 } } }
I20260620 06:03:39.418702 14813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [sys.catalog]: This master's current role is: LEADER
I20260620 06:03:39.418707 14812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834 [sys.catalog]: This master's current role is: LEADER
I20260620 06:03:39.419054 14820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:03:39.421991 14820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:03:39.422402 14806 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:03:39.431995 14820 catalog_manager.cc:1347] Generated new cluster ID: 05bf59586ff1445b8b8b17c2d6687c0d
I20260620 06:03:39.432093 14820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:03:39.451180 14820 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:03:39.452200 14820 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:03:39.462213 14820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834: Generated new TSK 0
I20260620 06:03:39.463172 14820 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:03:40.192274 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:38181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34691
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:03:40.317215 14851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:03:40.317575 14851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:03:40.317683 14851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:03:40.322557 14851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:03:40.322657 14851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:03:40.322767 14851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:03:40.328804 14851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.115.20:38181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:03:40.330039 14851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:03:40.331195 14851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:03:40.343240 14856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:40.343240 14859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:40.343341 14857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:03:40.343837 14851 server_base.cc:1034] running on GCE node
I20260620 06:03:40.344369 14851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:03:40.345235 14851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:03:40.346454 14851 hybrid_clock.cc:648] HybridClock initialized: now 1781935420346421 us; error 57 us; skew 500 ppm
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:03:40.355094 14851 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:03:40.356498 14851 webserver.cc:460] Webserver started at http://127.3.115.1:34181/ using document root <none> and password file <none>
I20260620 06:03:40.357324 14851 fs_manager.cc:359] Metadata directory not provided
I20260620 06:03:40.357429 14851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:03:40.357717 14851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:03:40.375036 14851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "498b0f504f434a4aac1a8a849f17b397"
format_stamp: "Formatted at 2026-06-20 06:03:40 on dist-test-slave-4h14"
server_key: "c500062b356b7d3a1ba0081531850a0d"
server_key_iv: "9748d96a1ee5871ce88a9dfb4375fb85"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:40.376205 14851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "498b0f504f434a4aac1a8a849f17b397"
format_stamp: "Formatted at 2026-06-20 06:03:40 on dist-test-slave-4h14"
server_key: "c500062b356b7d3a1ba0081531850a0d"
server_key_iv: "9748d96a1ee5871ce88a9dfb4375fb85"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:40.382851 14851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260620 06:03:40.418972 14868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:40.422031 14851 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260620 06:03:40.422281 14851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "498b0f504f434a4aac1a8a849f17b397"
format_stamp: "Formatted at 2026-06-20 06:03:40 on dist-test-slave-4h14"
server_key: "c500062b356b7d3a1ba0081531850a0d"
server_key_iv: "9748d96a1ee5871ce88a9dfb4375fb85"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:40.422521 14851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:03:40.449751 14851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:03:40.453789 14851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:03:40.454064 14851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:03:40.454823 14851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:03:40.455986 14851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:03:40.456099 14851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:40.456230 14851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:03:40.456316 14851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:40.467391 14851 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:43619
I20260620 06:03:40.471769 14851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:03:40.481727 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 14851
I20260620 06:03:40.481864 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:03:40.482182 3532 external_mini_cluster.cc:1428] Setting key ef2a2c011f415710318a223f1baf2027
I20260620 06:03:40.473659 14981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:43619 every 8 connection(s)
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:03:40.496085 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:40.475834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:34011 (local address 127.3.115.62:34691)
0620 06:03:40.476137 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:40.476142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:40.477979 (+ 1837us) server_negotiation.cc:407] Connection header received
0620 06:03:40.478034 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:40.478037 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:40.478104 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:40.478191 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:03:40.479780 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.480699 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.483661 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.486157 (+ 2496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.486264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:40.489810 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:03:40.489815 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:03:40.489818 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:03:40.489849 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:03:40.491864 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:40.492574 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:40.492578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:40.492580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:40.492724 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:40.494493 (+ 1769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:40.494497 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:40.494499 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:40.494767 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:03:40.494870 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:40.495727 (+ 857us) server_negotiation.cc:299] Negotiation successful
0620 06:03:40.495883 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
I20260620 06:03:40.498442 14984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:40.474626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34691 (local address 127.3.115.1:34011)
0620 06:03:40.476422 (+ 1796us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:40.476487 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:40.477643 (+ 1156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:40.478518 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:40.478528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:40.478910 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:03:40.479603 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:40.479622 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.480850 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.480854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:40.483523 (+ 2669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:40.483537 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.486523 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.486527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:40.486774 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:40.487545 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:03:40.487583 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:03:40.489544 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:03:40.492023 (+ 2479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:03:40.492030 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:03:40.492051 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:03:40.492429 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:03:40.494094 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:03:40.494098 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:03:40.494100 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:03:40.494342 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:03:40.494949 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:03:40.494957 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:03:40.495384 (+ 427us) client_negotiation.cc:769] Sending connection context
0620 06:03:40.496372 (+ 988us) client_negotiation.cc:240] Negotiation successful
0620 06:03:40.497405 (+ 1033us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1560,"thread_start_us":150,"threads_started":1}
I20260620 06:03:40.499905 14982 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34691
I20260620 06:03:40.500164 14982 heartbeater.cc:464] Registering TS with master...
I20260620 06:03:40.500779 14982 heartbeater.cc:511] Master 127.3.115.62:34691 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:03:40.502552 14770 ts_manager.cc:194] Registered new tserver with Master: 498b0f504f434a4aac1a8a849f17b397 (127.3.115.1:43619)
I20260620 06:03:40.503847 14770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:34011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:03:40.521610 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:38181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34691
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 06:03:40.692310 14990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:03:40.692795 14990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:03:40.692925 14990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:03:40.699294 14990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:03:40.699395 14990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:03:40.699487 14990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:03:40.707330 14990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.115.20:38181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:03:40.708812 14990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:03:40.709923 14990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:03:40.721258 14996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:40.721789 14999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:40.721858 14997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:03:40.723026 14990 server_base.cc:1034] running on GCE node
I20260620 06:03:40.723516 14990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:03:40.724362 14990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:03:40.725543 14990 hybrid_clock.cc:648] HybridClock initialized: now 1781935420725516 us; error 42 us; skew 500 ppm
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:03:40.729615 14990 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:03:40.730741 14990 webserver.cc:460] Webserver started at http://127.3.115.2:40009/ using document root <none> and password file <none>
I20260620 06:03:40.731432 14990 fs_manager.cc:359] Metadata directory not provided
I20260620 06:03:40.731532 14990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:03:40.731801 14990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:03:40.748381 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "13df44874dab4754b58a57d0e592d424"
format_stamp: "Formatted at 2026-06-20 06:03:40 on dist-test-slave-4h14"
server_key: "6ed9bb8269ca2864a248f1e182c7e995"
server_key_iv: "da4015d5362413d9033b641c50e9afcd"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:40.749334 14990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "13df44874dab4754b58a57d0e592d424"
format_stamp: "Formatted at 2026-06-20 06:03:40 on dist-test-slave-4h14"
server_key: "6ed9bb8269ca2864a248f1e182c7e995"
server_key_iv: "da4015d5362413d9033b641c50e9afcd"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:40.754906 14990 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260620 06:03:40.779091 15006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:40.781036 14990 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260620 06:03:40.781260 14990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "13df44874dab4754b58a57d0e592d424"
format_stamp: "Formatted at 2026-06-20 06:03:40 on dist-test-slave-4h14"
server_key: "6ed9bb8269ca2864a248f1e182c7e995"
server_key_iv: "da4015d5362413d9033b641c50e9afcd"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:40.781523 14990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:03:40.820744 14990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:03:40.824864 14990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:03:40.825091 14990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:03:40.825832 14990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:03:40.826980 14990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:03:40.827080 14990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:40.827163 14990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:03:40.827214 14990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:40.837739 14990 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:39541
I20260620 06:03:40.838944 14990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:03:40.842130 15119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:39541 every 8 connection(s)
I20260620 06:03:40.846570 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 14990
I20260620 06:03:40.846709 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:03:40.847015 3532 external_mini_cluster.cc:1428] Setting key 44f391a843e0024e8862dbcba8edc3bf
Jun 20 06:03:40 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:03:40.860110 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:40.844328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:54931 (local address 127.3.115.62:34691)
0620 06:03:40.844464 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:40.844468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:40.846360 (+ 1892us) server_negotiation.cc:407] Connection header received
0620 06:03:40.847709 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:40.847713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:40.847768 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:40.847823 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:03:40.849554 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.850406 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.852070 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.852439 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.852522 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:40.855717 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:03:40.855721 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:03:40.855724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:03:40.855758 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:03:40.857481 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:40.858131 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:40.858135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:40.858137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:40.858222 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:40.858685 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:40.858689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:40.858690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:40.858913 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:03:40.858994 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:40.859795 (+ 801us) server_negotiation.cc:299] Negotiation successful
0620 06:03:40.859938 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:03:40.862401 15122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:40.844885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34691 (local address 127.3.115.2:54931)
0620 06:03:40.846178 (+ 1293us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:40.846239 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:40.847448 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:40.848235 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:40.848244 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:40.848621 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:03:40.849349 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:40.849373 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.850592 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.850595 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:40.851929 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:40.851939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:40.852957 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:40.852962 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:40.853165 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:40.853845 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:03:40.853866 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:03:40.855517 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:03:40.857637 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:03:40.857643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:03:40.857661 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:03:40.858008 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:03:40.858325 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:03:40.858328 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:03:40.858330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:03:40.858567 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:03:40.859062 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:03:40.859069 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:03:40.859481 (+ 412us) client_negotiation.cc:769] Sending connection context
0620 06:03:40.860805 (+ 1324us) client_negotiation.cc:240] Negotiation successful
0620 06:03:40.861191 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1068,"thread_start_us":166,"threads_started":1}
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:03:40.864065 15120 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34691
I20260620 06:03:40.864328 15120 heartbeater.cc:464] Registering TS with master...
I20260620 06:03:40.865103 15120 heartbeater.cc:511] Master 127.3.115.62:34691 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:03:40.866392 14770 ts_manager.cc:194] Registered new tserver with Master: 13df44874dab4754b58a57d0e592d424 (127.3.115.2:39541)
I20260620 06:03:40.867223 14770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:54931
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:03:40.899849 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:38181/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34691
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:43385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:03:41 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
W20260620 06:03:41.067126 15125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:03:41.067569 15125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:03:41.067658 15125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:03:41.073752 15125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:03:41.073891 15125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:03:41.074008 15125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:03:41.081916 15125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:43385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.115.20:38181/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:34691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:03:41.083511 15125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:03:41.084975 15125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:03:41.098613 15125 server_base.cc:1034] running on GCE node
W20260620 06:03:41.099038 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:41.099400 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:03:41.099654 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:03:41.100231 15125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:03:41.101040 15125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:03:41.102198 15125 hybrid_clock.cc:648] HybridClock initialized: now 1781935421102127 us; error 89 us; skew 500 ppm
Jun 20 06:03:41 dist-test-slave-4h14 krb5kdc[14294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:03:41.105880 15125 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:03:41.107023 15125 webserver.cc:460] Webserver started at http://127.3.115.3:46019/ using document root <none> and password file <none>
I20260620 06:03:41.107757 15125 fs_manager.cc:359] Metadata directory not provided
I20260620 06:03:41.107851 15125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:03:41.108091 15125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:03:41 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:03:41.131927 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "f4a6505a64634560a36edf694bef0b44"
format_stamp: "Formatted at 2026-06-20 06:03:41 on dist-test-slave-4h14"
server_key: "c9a95ef28965f3023611ae9479de737c"
server_key_iv: "994994c630a525b5482907dd8b3575e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:41.132874 15125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "f4a6505a64634560a36edf694bef0b44"
format_stamp: "Formatted at 2026-06-20 06:03:41 on dist-test-slave-4h14"
server_key: "c9a95ef28965f3023611ae9479de737c"
server_key_iv: "994994c630a525b5482907dd8b3575e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:41.140924 15125 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.000s
I20260620 06:03:41.169436 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:41.171428 15125 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.001s sys 0.001s
I20260620 06:03:41.171654 15125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "f4a6505a64634560a36edf694bef0b44"
format_stamp: "Formatted at 2026-06-20 06:03:41 on dist-test-slave-4h14"
server_key: "c9a95ef28965f3023611ae9479de737c"
server_key_iv: "994994c630a525b5482907dd8b3575e9"
server_key_version: "kuduclusterkey@0"
I20260620 06:03:41.171912 15125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:03:41.192291 15125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:03:41.196264 15125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:03:41.196471 15125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:03:41.197270 15125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:03:41.198432 15125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:03:41.198511 15125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:41.198635 15125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:03:41.198691 15125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:03:41.210749 15125 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:39817
I20260620 06:03:41.212186 15125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:03:41.213217 15253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:39817 every 8 connection(s)
I20260620 06:03:41.222216 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 15125
I20260620 06:03:41.222352 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:03:41.222635 3532 external_mini_cluster.cc:1428] Setting key e38374d8a34fd9281c3b84be53f45956
Jun 20 06:03:41 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935421, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:03:41.230703 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.214447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:55503 (local address 127.3.115.62:34691)
0620 06:03:41.214596 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.214601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.216203 (+ 1602us) server_negotiation.cc:407] Connection header received
0620 06:03:41.217584 (+ 1381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.217588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.217648 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.217746 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:03:41.219468 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.220268 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.222139 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.222494 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.222588 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.226001 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:03:41.226006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:03:41.226009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:03:41.226039 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:03:41.227905 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:41.228596 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:41.228601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:41.228603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:41.228742 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:41.229299 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:41.229303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:41.229304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:41.229540 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:03:41.229635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.230396 (+ 761us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.230539 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:03:41.232153 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.214966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:34691 (local address 127.3.115.3:55503)
0620 06:03:41.216001 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.216076 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.217292 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.218085 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.218097 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.218491 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:03:41.219287 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.219310 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.220423 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.220427 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.221948 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.221964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.222836 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.222844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.223169 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.223879 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:03:41.223903 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:03:41.225799 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:03:41.228030 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:03:41.228037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:03:41.228066 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:03:41.228465 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:03:41.228844 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:03:41.228848 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:03:41.228850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:03:41.229181 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:03:41.229660 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:03:41.229667 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:03:41.230083 (+ 416us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.230318 (+ 235us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.230725 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":784,"thread_start_us":175,"threads_started":1}
I20260620 06:03:41.233608 15254 heartbeater.cc:347] Connected to a master server at 127.3.115.62:34691
I20260620 06:03:41.233901 15254 heartbeater.cc:464] Registering TS with master...
I20260620 06:03:41.234561 15254 heartbeater.cc:511] Master 127.3.115.62:34691 requested a full tablet report, sending...
I20260620 06:03:41.235868 14770 ts_manager.cc:194] Registered new tserver with Master: f4a6505a64634560a36edf694bef0b44 (127.3.115.3:39817)
I20260620 06:03:41.236732 14770 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:55503
I20260620 06:03:41.237449 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:03:41.252593 14985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.241134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33648 (local address 127.3.115.62:34691)
0620 06:03:41.241269 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.241274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.243008 (+ 1734us) server_negotiation.cc:407] Connection header received
0620 06:03:41.243135 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.243138 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.243200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.243305 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:03:41.244230 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.245109 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.246331 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.246691 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.246828 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.248622 (+ 1794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:03:41.248627 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:03:41.248630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:03:41.248706 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:03:41.250593 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:41.251187 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:41.251191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:41.251193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:41.251296 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:03:41.251616 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:03:41.251619 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:03:41.251621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:03:41.251859 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:03:41.251991 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.252289 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.252421 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:03:41.256976 14769 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:03:41.261785 14769 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:03:41.283907 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.274079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33376 (local address 127.3.115.2:39541)
0620 06:03:41.275191 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.275210 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.276846 (+ 1636us) server_negotiation.cc:407] Connection header received
0620 06:03:41.277201 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.277208 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.277468 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.277621 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.278827 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.280069 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.281927 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.282830 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.282926 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.283117 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.283586 (+ 469us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.283709 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":158,"threads_started":1}
I20260620 06:03:41.284430 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.273937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39541 (local address 127.0.0.1:33376)
0620 06:03:41.276646 (+ 2709us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.276742 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.276977 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.278096 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.278102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.278515 (+ 413us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.278676 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.278684 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.280221 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.280225 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.281749 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.281761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.283228 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.283232 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.283344 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.283387 (+ 43us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.284099 (+ 712us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.284260 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2502,"thread_start_us":135,"threads_started":1}
I20260620 06:03:41.290899 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.275384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.3.115.1:43619)
0620 06:03:41.275800 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.275808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.284999 (+ 9191us) server_negotiation.cc:407] Connection header received
0620 06:03:41.285266 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.285273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.285505 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.285648 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.286209 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.287286 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.288949 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.289879 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.289964 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.290139 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.290612 (+ 473us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.290734 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":115,"threads_started":1}
I20260620 06:03:41.290902 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.276202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39817 (local address 127.0.0.1:58242)
0620 06:03:41.284870 (+ 8668us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.284885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.284993 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.285831 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.285835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.285853 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.285976 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.285982 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.287448 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.287451 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.288819 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.288830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.290246 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.290249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.290421 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.290473 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.290549 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.290740 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8597,"mutex_wait_us":99,"spinlock_wait_cycles":23424,"thread_start_us":120,"threads_started":1}
I20260620 06:03:41.291355 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.275227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:43619 (local address 127.0.0.1:52782)
0620 06:03:41.284845 (+ 9618us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.284862 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.285081 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.285847 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.285850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.285870 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.286066 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.286072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.287438 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.287441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.288796 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.288810 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.290245 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.290249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.290422 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.290472 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.291139 (+ 667us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.291225 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9531,"mutex_wait_us":230,"thread_start_us":102,"threads_started":1}
I20260620 06:03:41.291546 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.276733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58242 (local address 127.3.115.3:39817)
0620 06:03:41.277190 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.277203 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.285119 (+ 7916us) server_negotiation.cc:407] Connection header received
0620 06:03:41.285220 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.285226 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.285494 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.285666 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.286128 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.287309 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.288991 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.289890 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.290009 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.290179 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.291288 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.291404 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":156,"threads_started":1}
I20260620 06:03:41.294726 14916 tablet_service.cc:1467] Processing CreateTablet for tablet 799ebfb72982444584ea227e29d30dca (DEFAULT_TABLE table=test-table [id=fb0dee87662b41b49b21acc253768e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:03:41.294726 15188 tablet_service.cc:1467] Processing CreateTablet for tablet 799ebfb72982444584ea227e29d30dca (DEFAULT_TABLE table=test-table [id=fb0dee87662b41b49b21acc253768e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:03:41.295951 14916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799ebfb72982444584ea227e29d30dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:03:41.295953 15188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799ebfb72982444584ea227e29d30dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:03:41.303277 15054 tablet_service.cc:1467] Processing CreateTablet for tablet 799ebfb72982444584ea227e29d30dca (DEFAULT_TABLE table=test-table [id=fb0dee87662b41b49b21acc253768e0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:03:41.304569 15054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 799ebfb72982444584ea227e29d30dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:03:41.304762 15272 tablet_bootstrap.cc:492] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Bootstrap starting.
I20260620 06:03:41.307883 15272 tablet_bootstrap.cc:654] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Neither blocks nor log segments found. Creating new log.
I20260620 06:03:41.310693 15272 log.cc:824] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Log is configured to *not* fsync() on all Append() calls
I20260620 06:03:41.313239 15272 tablet_bootstrap.cc:492] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: No bootstrap required, opened a new log
I20260620 06:03:41.313468 15272 ts_tablet_manager.cc:1397] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260620 06:03:41.319600 15272 raft_consensus.cc:348] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.320125 15272 raft_consensus.cc:374] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:03:41.320250 15272 raft_consensus.cc:729] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a6505a64634560a36edf694bef0b44, State: Initialized, Role: FOLLOWER
I20260620 06:03:41.320919 15272 consensus_queue.cc:260] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.322265 15272 ts_tablet_manager.cc:1428] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260620 06:03:41.322795 15254 heartbeater.cc:503] Master 127.3.115.62:34691 was elected leader, sending a full tablet report...
I20260620 06:03:41.323206 15274 tablet_bootstrap.cc:492] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: Bootstrap starting.
I20260620 06:03:41.327945 15275 tablet_bootstrap.cc:492] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: Bootstrap starting.
I20260620 06:03:41.330878 15275 tablet_bootstrap.cc:654] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: Neither blocks nor log segments found. Creating new log.
I20260620 06:03:41.331959 15274 tablet_bootstrap.cc:654] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: Neither blocks nor log segments found. Creating new log.
I20260620 06:03:41.331991 15275 log.cc:824] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: Log is configured to *not* fsync() on all Append() calls
I20260620 06:03:41.333082 15274 log.cc:824] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: Log is configured to *not* fsync() on all Append() calls
I20260620 06:03:41.335579 15274 tablet_bootstrap.cc:492] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: No bootstrap required, opened a new log
I20260620 06:03:41.335822 15274 ts_tablet_manager.cc:1397] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260620 06:03:41.340179 15274 raft_consensus.cc:348] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.340544 15274 raft_consensus.cc:374] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:03:41.340621 15274 raft_consensus.cc:729] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13df44874dab4754b58a57d0e592d424, State: Initialized, Role: FOLLOWER
I20260620 06:03:41.341251 15274 consensus_queue.cc:260] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.341480 15275 tablet_bootstrap.cc:492] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: No bootstrap required, opened a new log
I20260620 06:03:41.341742 15275 ts_tablet_manager.cc:1397] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260620 06:03:41.342541 15274 ts_tablet_manager.cc:1428] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260620 06:03:41.342964 15120 heartbeater.cc:503] Master 127.3.115.62:34691 was elected leader, sending a full tablet report...
W20260620 06:03:41.342962 15121 tablet.cc:2369] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:03:41.346850 15275 raft_consensus.cc:348] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.347164 15275 raft_consensus.cc:374] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:03:41.347226 15275 raft_consensus.cc:729] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498b0f504f434a4aac1a8a849f17b397, State: Initialized, Role: FOLLOWER
I20260620 06:03:41.347720 15275 consensus_queue.cc:260] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.348464 14982 heartbeater.cc:503] Master 127.3.115.62:34691 was elected leader, sending a full tablet report...
I20260620 06:03:41.348944 15275 ts_tablet_manager.cc:1428] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
Jun 20 06:03:41 dist-test-slave-4h14 krb5kdc[14294](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935420, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
W20260620 06:03:41.464075 15255 tablet.cc:2369] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:03:41.475047 14983 tablet.cc:2369] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:03:41.542316 15276 raft_consensus.cc:482] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:03:41.542583 15276 raft_consensus.cc:504] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.544148 15276 leader_election.cc:288] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13df44874dab4754b58a57d0e592d424 (127.3.115.2:39541), 498b0f504f434a4aac1a8a849f17b397 (127.3.115.1:43619)
I20260620 06:03:41.549293 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.544368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39541 (local address 127.3.115.3:44513)
0620 06:03:41.544792 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.544809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.544914 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.545374 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.545377 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.545405 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.545533 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.545540 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.546672 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.546675 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.547887 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.547896 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.548873 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.548877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.548972 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.549007 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.549073 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.549145 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332}
I20260620 06:03:41.549525 15280 raft_consensus.cc:482] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:03:41.549736 15280 raft_consensus.cc:504] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.549997 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.545498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:38907 (local address 127.3.115.1:43619)
0620 06:03:41.545740 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.545746 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.545969 (+ 223us) server_negotiation.cc:407] Connection header received
0620 06:03:41.546152 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.546156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.546205 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.546250 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.546615 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.547342 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.548761 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.549322 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.549403 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.549469 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.549823 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.549900 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260620 06:03:41.550522 15283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.544412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:43619 (local address 127.3.115.3:38907)
0620 06:03:41.545885 (+ 1473us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.545900 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.546037 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.546366 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.546369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.546382 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.546482 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.546499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.547479 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.547482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.548598 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.548607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.549596 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.549600 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.549686 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.549707 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.550171 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.550242 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"thread_start_us":141,"threads_started":1}
I20260620 06:03:41.551127 15280 leader_election.cc:288] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13df44874dab4754b58a57d0e592d424 (127.3.115.2:39541), f4a6505a64634560a36edf694bef0b44 (127.3.115.3:39817)
I20260620 06:03:41.551127 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799ebfb72982444584ea227e29d30dca" candidate_uuid: "f4a6505a64634560a36edf694bef0b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498b0f504f434a4aac1a8a849f17b397" is_pre_election: true
I20260620 06:03:41.551373 14924 raft_consensus.cc:2454] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a6505a64634560a36edf694bef0b44 in term 0.
I20260620 06:03:41.551862 15141 leader_election.cc:302] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 498b0f504f434a4aac1a8a849f17b397, f4a6505a64634560a36edf694bef0b44; no voters:
I20260620 06:03:41.553576 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.544744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:44513 (local address 127.3.115.2:39541)
0620 06:03:41.545105 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.545111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.545125 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:03:41.545179 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.545182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.545230 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.545315 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.545802 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.546555 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.548048 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.548623 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.548750 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.548819 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.553365 (+ 4546us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.553465 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260620 06:03:41.554458 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799ebfb72982444584ea227e29d30dca" candidate_uuid: "f4a6505a64634560a36edf694bef0b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13df44874dab4754b58a57d0e592d424" is_pre_election: true
I20260620 06:03:41.554553 15276 raft_consensus.cc:2781] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:03:41.554662 15276 raft_consensus.cc:482] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:03:41.554718 15276 raft_consensus.cc:3037] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:03:41.554781 15074 raft_consensus.cc:2454] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f4a6505a64634560a36edf694bef0b44 in term 0.
I20260620 06:03:41.556567 15276 raft_consensus.cc:504] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.557333 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799ebfb72982444584ea227e29d30dca" candidate_uuid: "f4a6505a64634560a36edf694bef0b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13df44874dab4754b58a57d0e592d424"
I20260620 06:03:41.557462 15074 raft_consensus.cc:3037] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:03:41.558360 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.551820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:36499 (local address 127.3.115.2:39541)
0620 06:03:41.553746 (+ 1926us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.553750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.553762 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:03:41.553798 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.553801 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.553839 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.553913 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.554524 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.555176 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.557120 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.557664 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.557755 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.557819 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.558162 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.558235 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1870,"thread_start_us":65,"threads_started":1}
I20260620 06:03:41.558857 15284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.551463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39541 (local address 127.3.115.1:36499)
0620 06:03:41.552300 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.552314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.552407 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.554278 (+ 1871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.554281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.554297 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.554403 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.554408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.555319 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.555322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.556973 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.556983 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.557914 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.557918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.558022 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.558046 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.558557 (+ 511us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.558640 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":122,"threads_started":1}
I20260620 06:03:41.558862 15074 raft_consensus.cc:2454] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a6505a64634560a36edf694bef0b44 in term 1.
I20260620 06:03:41.559744 15142 leader_election.cc:302] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13df44874dab4754b58a57d0e592d424, f4a6505a64634560a36edf694bef0b44; no voters:
I20260620 06:03:41.560132 15074 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799ebfb72982444584ea227e29d30dca" candidate_uuid: "498b0f504f434a4aac1a8a849f17b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13df44874dab4754b58a57d0e592d424" is_pre_election: true
I20260620 06:03:41.560319 15074 raft_consensus.cc:2379] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 498b0f504f434a4aac1a8a849f17b397 in current term 1: Already voted for candidate f4a6505a64634560a36edf694bef0b44 in this term.
I20260620 06:03:41.561097 15276 leader_election.cc:288] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [CANDIDATE]: Term 1 election: Requested vote from peers 13df44874dab4754b58a57d0e592d424 (127.3.115.2:39541), 498b0f504f434a4aac1a8a849f17b397 (127.3.115.1:43619)
I20260620 06:03:41.561264 15276 raft_consensus.cc:2781] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:03:41.561378 15276 raft_consensus.cc:686] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 1 LEADER]: Becoming Leader. State: Replica: f4a6505a64634560a36edf694bef0b44, State: Running, Role: LEADER
I20260620 06:03:41.561746 14924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799ebfb72982444584ea227e29d30dca" candidate_uuid: "f4a6505a64634560a36edf694bef0b44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498b0f504f434a4aac1a8a849f17b397"
I20260620 06:03:41.561769 15276 consensus_queue.cc:237] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } }
I20260620 06:03:41.561877 14924 raft_consensus.cc:3037] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:03:41.563212 14924 raft_consensus.cc:2454] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f4a6505a64634560a36edf694bef0b44 in term 1.
I20260620 06:03:41.564313 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.552882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:40483 (local address 127.3.115.3:39817)
0620 06:03:41.552982 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.552986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.553007 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:03:41.553046 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.553048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.553082 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.553168 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:03:41.553437 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.554163 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.560632 (+ 6469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.561222 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.561309 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.561380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.564115 (+ 2735us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.564201 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:03:41.565438 14769 catalog_manager.cc:5495] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 reported cstate change: term changed from 0 to 1, leader changed from <none> to f4a6505a64634560a36edf694bef0b44 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "f4a6505a64634560a36edf694bef0b44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4a6505a64634560a36edf694bef0b44" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39817 } health_report { overall_health: HEALTHY } } }
I20260620 06:03:41.566908 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.552502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39817 (local address 127.3.115.1:40483)
0620 06:03:41.552777 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.552789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:03:41.552873 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:03:41.553178 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:03:41.553181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:03:41.553196 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:03:41.553319 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.553329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.559291 (+ 5962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.559294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.560478 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:03:41.560497 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.563855 (+ 3358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.563858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:03:41.563964 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.563989 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:03:41.566664 (+ 2675us) client_negotiation.cc:240] Negotiation successful
0620 06:03:41.566761 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":136,"threads_started":1}
I20260620 06:03:41.567956 15208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "799ebfb72982444584ea227e29d30dca" candidate_uuid: "498b0f504f434a4aac1a8a849f17b397" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4a6505a64634560a36edf694bef0b44" is_pre_election: true
I20260620 06:03:41.569201 14872 leader_election.cc:302] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 498b0f504f434a4aac1a8a849f17b397; no voters: 13df44874dab4754b58a57d0e592d424, f4a6505a64634560a36edf694bef0b44
I20260620 06:03:41.569772 15280 raft_consensus.cc:2726] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 06:03:41.594426 15271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.589051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.3.115.3:39817)
0620 06:03:41.589181 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:03:41.589187 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:03:41.589203 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:03:41.589425 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:03:41.589429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:03:41.589504 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:03:41.589607 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:03:41.590010 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.590802 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.592059 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:03:41.592422 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:03:41.592520 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:03:41.592987 (+ 467us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:03:41.593600 (+ 613us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:03:41.593739 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 06:03:41.594207 (+ 468us) server_negotiation.cc:299] Negotiation successful
0620 06:03:41.594273 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:03:41.602118 14924 raft_consensus.cc:1261] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 1 FOLLOWER]: Refusing update from remote peer f4a6505a64634560a36edf694bef0b44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:03:41.602109 15074 raft_consensus.cc:1261] T 799ebfb72982444584ea227e29d30dca P 13df44874dab4754b58a57d0e592d424 [term 1 FOLLOWER]: Refusing update from remote peer f4a6505a64634560a36edf694bef0b44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:03:41.602887 15287 consensus_queue.cc:1035] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498b0f504f434a4aac1a8a849f17b397" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 43619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:03:41.603376 15287 consensus_queue.cc:1035] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13df44874dab4754b58a57d0e592d424" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:03:41.617034 15290 mvcc.cc:205] Tried to move back new op lower bound from 7298807486873288704 to 7298807486720008192. Current Snapshot: MvccSnapshot[applied={T|T < 7298807486873288704}]
I20260620 06:03:41.622023 15291 mvcc.cc:205] Tried to move back new op lower bound from 7298807486873288704 to 7298807486720008192. Current Snapshot: MvccSnapshot[applied={T|T < 7298807486873288704}]
I20260620 06:03:41.636029 15292 mvcc.cc:205] Tried to move back new op lower bound from 7298807486873288704 to 7298807486720008192. Current Snapshot: MvccSnapshot[applied={T|T < 7298807486873288704 or (T in {7298807486873288704})}]
I20260620 06:03:41.652369 14770 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:03:41.652598 14770 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:03:41.662978 14770 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 366deb5856e944dcb2ca9697c103d834: Sending DeleteTablet for 3 replicas of tablet 799ebfb72982444584ea227e29d30dca
I20260620 06:03:41.664150 15053 tablet_service.cc:1514] Processing DeleteTablet for tablet 799ebfb72982444584ea227e29d30dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:03:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:33376
I20260620 06:03:41.664057 15054 tablet_service.cc:1514] Processing DeleteTablet for tablet 799ebfb72982444584ea227e29d30dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:03:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:33376
I20260620 06:03:41.664444 14916 tablet_service.cc:1514] Processing DeleteTablet for tablet 799ebfb72982444584ea227e29d30dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:03:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:52782
I20260620 06:03:41.665302 15188 tablet_service.cc:1514] Processing DeleteTablet for tablet 799ebfb72982444584ea227e29d30dca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:03:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:58242
I20260620 06:03:41.666199 15300 tablet_replica.cc:330] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397: stopping tablet replica
I20260620 06:03:41.666580 15300 raft_consensus.cc:2229] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:03:41.666936 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 14851
I20260620 06:03:41.667030 15300 raft_consensus.cc:2258] T 799ebfb72982444584ea227e29d30dca P 498b0f504f434a4aac1a8a849f17b397 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 06:03:41.678541 14754 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:43619 (error 108)
I20260620 06:03:41.679793 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 14990
I20260620 06:03:41.688920 15301 tablet_replica.cc:330] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: stopping tablet replica
I20260620 06:03:41.690145 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.689778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:43619 (local address 127.0.0.1:48772)
0620 06:03:41.689945 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.690041 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:43619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 06:03:41.690379 14754 catalog_manager.cc:4584] TS 498b0f504f434a4aac1a8a849f17b397 (127.3.115.1:43619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 799ebfb72982444584ea227e29d30dca: Network error: Client connection negotiation failed: client connection to 127.3.115.1:43619: connect: Connection refused (error 111)
I20260620 06:03:41.690585 15301 raft_consensus.cc:2229] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:03:41.691112 15301 raft_consensus.cc:2258] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:03:41.694446 15301 ts_tablet_manager.cc:1905] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:03:41.697588 15301 ts_tablet_manager.cc:1918] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:03:41.697793 15301 log.cc:1196] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/799ebfb72982444584ea227e29d30dca
I20260620 06:03:41.698223 15301 ts_tablet_manager.cc:1939] T 799ebfb72982444584ea227e29d30dca P f4a6505a64634560a36edf694bef0b44: Deleting consensus metadata
I20260620 06:03:41.699896 14757 catalog_manager.cc:4841] TS f4a6505a64634560a36edf694bef0b44 (127.3.115.3:39817): tablet 799ebfb72982444584ea227e29d30dca (table test-table [id=fb0dee87662b41b49b21acc253768e0d]) successfully deleted
I20260620 06:03:41.709487 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:03:41.709083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39541 (local address 127.0.0.1:44218)
0620 06:03:41.709301 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 06:03:41.709364 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.2:39541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260620 06:03:41.712862 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 15125
I20260620 06:03:41.719857 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 14738
I20260620 06:03:41.731709 3532 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:03:42.067369 3532 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:03:42.067492 3532 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:03:42.413154 3532 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:03:42.413 UTC [14324] LOG: received smart shutdown request
2026-06-20 06:03:42.417 UTC [14324] LOG: background worker "logical replication launcher" (PID 14333) exited with exit code 1
2026-06-20 06:03:42.417 UTC [14328] LOG: shutting down
2026-06-20 06:03:42.593 UTC [14324] LOG: database system is shut down
2026-06-20T06:03:42Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (48421 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:03:42 dist-test-slave-4h14 krb5kdc[15313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:03:42 dist-test-slave-4h14 krb5kdc[15313](info): set up 2 sockets
Jun 20 06:03:42 dist-test-slave-4h14 krb5kdc[15313](info): commencing operation
krb5kdc: starting...
W20260620 06:03:43.756951 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:03:43 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935423, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:03:43Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/rangeradmin_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/rangeradmin_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/rangerlookup_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/rangerlookup_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
I20260620 06:03:43.935619 3532 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskVS9eKZ/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/postgres -l logfile start
2026-06-20 06:03:46.505 UTC [15343] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 06:03:46.505 UTC [15343] LOG: listening on IPv4 address "127.3.115.20", port 53901
2026-06-20 06:03:46.510 UTC [15343] LOG: listening on Unix socket "/tmp/.s.PGSQL.53901"
2026-06-20 06:03:46.545 UTC [15346] LOG: database system was shut down at 2026-06-20 06:03:45 UTC
2026-06-20 06:03:46.550 UTC [15343] LOG: database system is ready to accept connections
I20260620 06:03:47.524758 3532 mini_postgres.cc:96] Postgres bound to 53901
2026-06-20 06:03:47.531 UTC [15354] FATAL: database "slave" does not exist
127.3.115.20:53901 - accepting connections
I20260620 06:03:47.531774 3532 mini_ranger.cc:161] Starting Ranger...
I20260620 06:03:47.547463 3532 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 06:03:48.062098 3532 mini_ranger.cc:87] Created ranger Postgres database
I20260620 06:03:48.062168 3532 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-admin
2026-06-20 06:03:48,344 [I] DB FLAVOR :POSTGRES
2026-06-20 06:03:48,344 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:03:48,345 [I] Checking connection..
2026-06-20 06:03:48,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 06:03:48,596 [I] Checking connection passed.
2026-06-20 06:03:48,597 [I] --------- Verifying version history table ---------
2026-06-20 06:03:48,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:03:48,857 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:03:48,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:03:49,104 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 06:03:49,105 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 06:03:49,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 06:03:49.363 UTC [15418] WARNING: there is no transaction in progress
2026-06-20 06:03:49,370 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 06:03:49,371 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 06:03:49,645 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 06:03:49,646 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:03:49,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 06:03:49,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 06:03:50,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-4h14.c.gcp-upstream.internal','N') ;"
2026-06-20 06:03:50,412 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:03:50,656 [I] Table x_portal_user does not exist in database ranger
2026-06-20 06:03:50,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:03:50,905 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 06:03:50,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:03:51,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:03:51,385 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 06:03:51,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 06:03:52.437 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.455 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.473 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.489 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.509 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.528 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.591 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.598 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.615 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.628 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.642 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.659 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.671 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.679 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.688 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:52.696 UTC [15526] WARNING: there is no transaction in progress
2026-06-20 06:03:53,422 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 06:03:53,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 06:03:53,680 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 06:03:53,681 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 06:03:53,956 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 06:03:53,957 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 06:03:54,203 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 06:03:54,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 06:03:54,442 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 06:03:54,677 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-4h14.c.gcp-upstream.internal';"
2026-06-20 06:03:54,909 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 06:03:54,909 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 06:03:55,178 [I] DB_PATCHES have already been applied
I20260620 06:03:55.184603 3532 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-admin:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskVS9eKZ/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 06:03:55.184731 3532 mini_ranger.cc:193] Using host: 127.3.115.20
I20260620 06:03:55.188227 3532 mini_ranger.cc:227] Ranger admin URL: http://127.3.115.20:40881
Jun 20, 2026 6:03:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:03:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:03:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 6:03:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 6:03:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 6:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 6:03:56 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 06:03:56.348093 3532 mini_ranger.cc:160] Time spent starting Ranger: real 8.816s user 0.000s sys 0.011s
Jun 20, 2026 6:04:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 6:04:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 20, 2026 6:04:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 6:04:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 6:04:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 06:04:18.715917 3532 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.115.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.115.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/rangerkms_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.115.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/rangerkms_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.115.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.115.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.115.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/HTTP_127.3.115.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 06:04:18 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 06:04:18.816987 3532 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 06:04:18.828346 3532 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 06:04:19.722761 3532 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 06:04:19.722831 3532 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms
I20260620 06:04:19.722846 3532 mini_ranger_kms.cc:227] Using postgres at 127.3.115.20:53901
2026-06-20 06:04:19,901 [I] DB FLAVOR :POSTGRES
2026-06-20 06:04:19,902 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 06:04:19,902 [I] Checking connection
2026-06-20 06:04:19,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 06:04:20,143 [I] connection success
2026-06-20 06:04:20,144 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 06:04:20,144 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 06:04:20,144 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 06:04:20,415 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 06:04:20,415 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 06:04:20,415 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 06:04:20,415 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.115.20:53901/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 06:04:20,674 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 06:04:21.179139 3532 mini_ranger_kms.cc:310] Created kms service
I20260620 06:04:21.393632 3532 mini_ranger_kms.cc:326] Created kudu user
I20260620 06:04:21.441352 3532 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 06:04:21 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:21.847394 3532 mini_ranger_kms.cc:384] Added ranger policy
I20260620 06:04:21.847555 3532 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms:/tmp/dist-test-taskVS9eKZ/build/debug/bin/postgresql.jar:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskVS9eKZ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 06:04:21.847582 3532 mini_ranger_kms.cc:240] Using host: 127.3.115.20
I20260620 06:04:21.851392 3532 mini_ranger_kms.cc:276] Ranger KMS PID: 15706
I20260620 06:04:21.851521 3532 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.115.20:45231
Jun 20, 2026 6:04:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 6:04:22 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 6:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 6:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 6:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 6:04:23 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 06:04:24.123061 3532 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.306s user 0.002s sys 0.160s
I20260620 06:04:24.123234 3532 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 06:04:24.123332 3532 mini_ranger_kms.cc:401] 127.3.115.20:45231/kms/v1/keys
Jun 20 06:04:24 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.115.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 06:04:24 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.115.20@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
Jun 20 06:04:25 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935464, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.115.20@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
Jun 20 06:04:27 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:04:27.453707 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45875
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:45875
--ranger_config_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:27.569000 15757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:27.569293 15757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:27.569334 15757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:27.573565 15757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:27.573650 15757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:27.573669 15757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:27.573686 15757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:27.573701 15757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:27.579362 15757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.115.20:45231/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:45875
--ranger_config_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:27.580541 15757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:27.581668 15757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:04:27.590853 15757 server_base.cc:1034] running on GCE node
W20260620 06:04:27.590756 15763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:27.590765 15762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:27.590791 15765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:27.591691 15757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:27.592942 15757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:27.594135 15757 hybrid_clock.cc:648] HybridClock initialized: now 1781935467594126 us; error 37 us; skew 500 ppm
Jun 20 06:04:27 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:27.600988 15757 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:27.602087 15757 webserver.cc:460] Webserver started at http://127.3.115.62:46217/ using document root <none> and password file <none>
I20260620 06:04:27.602700 15757 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:27.602782 15757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:27.603034 15757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:04:27 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935467, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:27.652056 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "43cc8d1e440b42bbaf57fc033f83075f"
format_stamp: "Formatted at 2026-06-20 06:04:27 on dist-test-slave-4h14"
server_key: "ab7ee0847e4983668a4ff2a186f9d5dc"
server_key_iv: "7702affef7da5e3a9b7a20e9bf51b0d0"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:27.652812 15757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "43cc8d1e440b42bbaf57fc033f83075f"
format_stamp: "Formatted at 2026-06-20 06:04:27 on dist-test-slave-4h14"
server_key: "ab7ee0847e4983668a4ff2a186f9d5dc"
server_key_iv: "7702affef7da5e3a9b7a20e9bf51b0d0"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:27.657083 15757 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:04:27.675274 15772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:27.676872 15757 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:27.677029 15757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "43cc8d1e440b42bbaf57fc033f83075f"
format_stamp: "Formatted at 2026-06-20 06:04:27 on dist-test-slave-4h14"
server_key: "ab7ee0847e4983668a4ff2a186f9d5dc"
server_key_iv: "7702affef7da5e3a9b7a20e9bf51b0d0"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:27.677170 15757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:27.698838 15757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:27.702159 15757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:27.702378 15757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:27.709781 15757 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:45875
I20260620 06:04:27.709789 15824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:45875 every 8 connection(s)
I20260620 06:04:27.710804 15757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:27.712261 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 15757
I20260620 06:04:27.712416 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:04:27.712740 3532 external_mini_cluster.cc:1428] Setting key 8154caae5463a94ca065d88bacd3fff6
I20260620 06:04:27.714016 15825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:27.720935 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Bootstrap starting.
Jun 20 06:04:27 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:27.723735 15825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:27.725154 15825 log.cc:824] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:27.727524 15825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: No bootstrap required, opened a new log
I20260620 06:04:27.728713 15828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:27.714247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.3.115.62:45875)
0620 06:04:27.714670 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:27.714689 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:27.714733 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:04:27.715581 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:27.715603 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:27.716031 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:27.716421 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:27.717244 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:27.718740 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:27.719637 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:27.720324 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:27.720485 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:27.723408 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:27.723414 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:27.723432 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:27.723463 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:27.725471 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:27.726051 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:27.726057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:27.726063 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:27.726190 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:27.726478 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:27.726481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:27.726483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:27.727020 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:27.727213 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:27.727468 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 06:04:27.727756 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":118,"threads_started":1}
I20260620 06:04:27.731304 15825 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:27.731545 15825 raft_consensus.cc:374] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:27.731583 15825 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cc8d1e440b42bbaf57fc033f83075f, State: Initialized, Role: FOLLOWER
I20260620 06:04:27.731986 15825 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:27.732111 15825 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:27.732152 15825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:27.732224 15825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:27.733289 15825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:27.733650 15825 leader_election.cc:302] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43cc8d1e440b42bbaf57fc033f83075f; no voters:
I20260620 06:04:27.733932 15825 leader_election.cc:288] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:04:27.734086 15830 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:27.734382 15830 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 LEADER]: Becoming Leader. State: Replica: 43cc8d1e440b42bbaf57fc033f83075f, State: Running, Role: LEADER
I20260620 06:04:27.734732 15830 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:27.735242 15825 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:27.736616 15831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "43cc8d1e440b42bbaf57fc033f83075f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } } }
I20260620 06:04:27.736637 15832 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43cc8d1e440b42bbaf57fc033f83075f. Latest consensus state: current_term: 1 leader_uuid: "43cc8d1e440b42bbaf57fc033f83075f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } } }
I20260620 06:04:27.736795 15831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:27.736802 15832 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:27.737229 15839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:27.737773 15825 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:04:27.742729 15839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:27.748997 15839 catalog_manager.cc:1347] Generated new cluster ID: 543038680c814ed2a01e5de00ecdc8f9
I20260620 06:04:27.749091 15839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:27.761101 15839 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:04:27.762107 15839 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:27.768833 15839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Generated new TSK 0
I20260620 06:04:27.769577 15839 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:04:28.472128 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45875
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:28.617251 15870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:28.617769 15870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:28.617880 15870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:28.623286 15870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:28.623381 15870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:28.623473 15870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:04:28.629513 15870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.115.20:45231/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:28.630785 15870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:28.632380 15870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:28.644541 15880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:28.644573 15877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:28.645062 15878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:28.645277 15870 server_base.cc:1034] running on GCE node
I20260620 06:04:28.645788 15870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:28.646687 15870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:28.647882 15870 hybrid_clock.cc:648] HybridClock initialized: now 1781935468647865 us; error 40 us; skew 500 ppm
Jun 20 06:04:28 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:28.651064 15870 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:04:28.652194 15870 webserver.cc:460] Webserver started at http://127.3.115.1:34851/ using document root <none> and password file <none>
I20260620 06:04:28.652918 15870 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:28.653012 15870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:28.653254 15870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:04:28 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:28.669052 15870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "68ccb4ff431a463d971437f998efe0a6"
format_stamp: "Formatted at 2026-06-20 06:04:28 on dist-test-slave-4h14"
server_key: "f53666aaaf68865aceaf76866001632d"
server_key_iv: "aaae51b2e4ea939d16c6f96a3c473eb4"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:28.669930 15870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "68ccb4ff431a463d971437f998efe0a6"
format_stamp: "Formatted at 2026-06-20 06:04:28 on dist-test-slave-4h14"
server_key: "f53666aaaf68865aceaf76866001632d"
server_key_iv: "aaae51b2e4ea939d16c6f96a3c473eb4"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:28.675416 15870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260620 06:04:28.695056 15887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:28.696592 15870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:04:28.696779 15870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "68ccb4ff431a463d971437f998efe0a6"
format_stamp: "Formatted at 2026-06-20 06:04:28 on dist-test-slave-4h14"
server_key: "f53666aaaf68865aceaf76866001632d"
server_key_iv: "aaae51b2e4ea939d16c6f96a3c473eb4"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:28.696929 15870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:28.724072 15870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:28.728937 15870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:28.729161 15870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:28.729789 15870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:28.730877 15870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:28.730954 15870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:28.731012 15870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:28.731037 15870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:28.741935 15870 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:39491
I20260620 06:04:28.741995 16002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:39491 every 8 connection(s)
I20260620 06:04:28.743237 15870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:04:28.751603 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 15870
I20260620 06:04:28.751770 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:04:28.752132 3532 external_mini_cluster.cc:1428] Setting key df1c4c808542ac70e4855cac4a2b4907
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Jun 20 06:04:28 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:04:28.775182 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:28.745486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:39899 (local address 127.3.115.62:45875)
0620 06:04:28.748968 (+ 3482us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:28.748978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:28.750593 (+ 1615us) server_negotiation.cc:407] Connection header received
0620 06:04:28.751268 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:28.751271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:28.751386 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:28.751490 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:28.760789 (+ 9299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:28.761655 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:28.763694 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:28.764072 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:28.764176 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:28.767994 (+ 3818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:28.767998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:28.768001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:28.768040 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:28.770160 (+ 2120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:28.771235 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:28.771239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:28.771241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:28.771350 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:28.771844 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:28.771848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:28.771850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:28.772114 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:28.772208 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:28.774816 (+ 2608us) server_negotiation.cc:299] Negotiation successful
0620 06:04:28.774978 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3376,"thread_start_us":127,"threads_started":1}
I20260620 06:04:28.777146 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:28.746103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45875 (local address 127.3.115.1:39899)
0620 06:04:28.749294 (+ 3191us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:28.749365 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:28.750988 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:28.751731 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:28.751743 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:28.752142 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:28.760453 (+ 8311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:28.760496 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:28.761872 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:28.761876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:28.763506 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:28.763520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:28.764441 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:28.764444 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:28.764698 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:28.765480 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:28.765504 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:28.767727 (+ 2223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:28.770661 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:28.770669 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:28.770690 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:28.771077 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:28.771477 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:28.771481 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:28.771484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:28.771707 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:28.772284 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:28.772291 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:28.772713 (+ 422us) client_negotiation.cc:769] Sending connection context
0620 06:04:28.775404 (+ 2691us) client_negotiation.cc:240] Negotiation successful
0620 06:04:28.775830 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2909,"thread_start_us":136,"threads_started":1}
I20260620 06:04:28.778774 16003 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45875
I20260620 06:04:28.779044 16003 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:28.779657 16003 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:28.781522 15789 ts_manager.cc:194] Registered new tserver with Master: 68ccb4ff431a463d971437f998efe0a6 (127.3.115.1:39491)
I20260620 06:04:28.782822 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:39899
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:04:28.787670 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45875
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:04:28 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 06:04:28.959767 16009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:28.960208 16009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:28.960278 16009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:28.966965 16009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:28.967080 16009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:28.967187 16009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:04:28.976167 16009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.115.20:45231/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:28.977826 16009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:28.979417 16009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:28.994282 16016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:28.994716 16018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:28.996078 16009 server_base.cc:1034] running on GCE node
W20260620 06:04:28.997624 16015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:28.998275 16009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:28.999183 16009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:29.000722 16009 hybrid_clock.cc:648] HybridClock initialized: now 1781935469000638 us; error 102 us; skew 500 ppm
Jun 20 06:04:28 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:29.006853 16009 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:04:29.008342 16009 webserver.cc:460] Webserver started at http://127.3.115.2:42027/ using document root <none> and password file <none>
I20260620 06:04:29.009316 16009 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:29.009425 16009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:29.009675 16009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:29.028352 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "ed289f750456487ab34f436dd43bcdba"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "c2e53fcc09b79c6b17723a15bfd6bf3f"
server_key_iv: "e37b61dadcbda801dbba9980232b7ce9"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.029209 16009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "ed289f750456487ab34f436dd43bcdba"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "c2e53fcc09b79c6b17723a15bfd6bf3f"
server_key_iv: "e37b61dadcbda801dbba9980232b7ce9"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.036134 16009 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260620 06:04:29.056177 16025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.066046 16009 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.002s sys 0.001s
I20260620 06:04:29.066246 16009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "ed289f750456487ab34f436dd43bcdba"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "c2e53fcc09b79c6b17723a15bfd6bf3f"
server_key_iv: "e37b61dadcbda801dbba9980232b7ce9"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.066437 16009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:29.088776 16009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:29.092981 16009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:29.093230 16009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:29.094017 16009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:29.095160 16009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:29.095240 16009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.095324 16009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:29.095356 16009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.105855 16009 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:42487
I20260620 06:04:29.107093 16009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:04:29.113617 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16009
I20260620 06:04:29.113788 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:04:29.114161 3532 external_mini_cluster.cc:1428] Setting key e8cf15e6239db6413d58103f95fc9515
I20260620 06:04:29.114586 16138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:42487 every 8 connection(s)
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:29.136099 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:29.119640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:44827 (local address 127.3.115.62:45875)
0620 06:04:29.119762 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:29.119766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:29.119849 (+ 83us) server_negotiation.cc:407] Connection header received
0620 06:04:29.121192 (+ 1343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:29.121197 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:29.121255 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:29.121311 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:29.123083 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.123928 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.125721 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.126091 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.126179 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:29.131235 (+ 5056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:29.131239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:29.131242 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:29.131280 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:29.133239 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:29.133988 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:29.133992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:29.133994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:29.134089 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:29.134556 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:29.134560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:29.134562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:29.134831 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:29.134921 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:29.135771 (+ 850us) server_negotiation.cc:299] Negotiation successful
0620 06:04:29.135921 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:04:29.137975 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:29.118442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45875 (local address 127.3.115.2:44827)
0620 06:04:29.119675 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:29.119738 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:29.120918 (+ 1180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:29.121771 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:29.121781 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:29.122158 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:29.122892 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:29.122915 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.124093 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.124096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:29.125570 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:29.125582 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.126398 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.126402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:29.126617 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:29.127323 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:29.127344 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:29.130956 (+ 3612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:29.133404 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:29.133411 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:29.133432 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:29.133851 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:29.134213 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:29.134216 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:29.134218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:29.134437 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:29.134995 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:29.135002 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:29.135390 (+ 388us) client_negotiation.cc:769] Sending connection context
0620 06:04:29.136312 (+ 922us) client_negotiation.cc:240] Negotiation successful
0620 06:04:29.136776 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":959,"thread_start_us":163,"threads_started":1}
I20260620 06:04:29.139573 16139 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45875
I20260620 06:04:29.139854 16139 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:29.140455 16139 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:29.141744 15789 ts_manager.cc:194] Registered new tserver with Master: ed289f750456487ab34f436dd43bcdba (127.3.115.2:42487)
I20260620 06:04:29.142551 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:44827
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:04:29.152611 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45875
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:29.337787 16144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:29.338196 16144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:29.338268 16144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:29.344650 16144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:29.344774 16144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:29.344861 16144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:04:29.353469 16144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.115.20:45231/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:29.354979 16144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:29.356316 16144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:29.381278 16150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:29.381892 16149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:29.383531 16152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:29.388092 16144 server_base.cc:1034] running on GCE node
I20260620 06:04:29.388854 16144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:29.389798 16144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:29.391180 16144 hybrid_clock.cc:648] HybridClock initialized: now 1781935469391100 us; error 100 us; skew 500 ppm
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:29.395368 16144 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:04:29.396648 16144 webserver.cc:460] Webserver started at http://127.3.115.3:39835/ using document root <none> and password file <none>
I20260620 06:04:29.397516 16144 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:29.397680 16144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:29.397990 16144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935468, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:29.422220 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "0986b21987b9480595f48e6685e7ad6c"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "6f630621630562b448ed5d7d3ea4660d"
server_key_iv: "a8e2b7d7b92e260641cebaebf4791207"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.423020 16144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "0986b21987b9480595f48e6685e7ad6c"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "6f630621630562b448ed5d7d3ea4660d"
server_key_iv: "a8e2b7d7b92e260641cebaebf4791207"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.434399 16144 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.004s
I20260620 06:04:29.481235 16159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.483325 16144 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.003s sys 0.000s
I20260620 06:04:29.483542 16144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "0986b21987b9480595f48e6685e7ad6c"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "6f630621630562b448ed5d7d3ea4660d"
server_key_iv: "a8e2b7d7b92e260641cebaebf4791207"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.483776 16144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:29.499305 16144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:29.503029 16144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:29.503284 16144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:29.504004 16144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:29.505110 16144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:29.505218 16144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.505313 16144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:29.505371 16144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.515520 16144 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:35097
I20260620 06:04:29.516884 16144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:04:29.524895 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16144
I20260620 06:04:29.525030 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:04:29.525389 3532 external_mini_cluster.cc:1428] Setting key 45492c0b492f489e62c77757148e4c27
I20260620 06:04:29.526029 16272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:35097 every 8 connection(s)
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:29.562537 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:29.531038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:51981 (local address 127.3.115.62:45875)
0620 06:04:29.532058 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:29.532063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:29.539785 (+ 7722us) server_negotiation.cc:407] Connection header received
0620 06:04:29.541219 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:29.541223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:29.541280 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:29.541331 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:29.551487 (+ 10156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.552324 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.554343 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.554683 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.554764 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:29.558065 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:29.558069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:29.558072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:29.558104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:29.559913 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:29.560594 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:29.560598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:29.560600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:29.560725 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:29.561151 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:29.561154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:29.561156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:29.561378 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:29.561455 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:29.562212 (+ 757us) server_negotiation.cc:299] Negotiation successful
0620 06:04:29.562346 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":925}
I20260620 06:04:29.564344 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:29.530926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45875 (local address 127.3.115.3:51981)
0620 06:04:29.539566 (+ 8640us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:29.539683 (+ 117us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:29.540940 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:29.541823 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:29.541834 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:29.542257 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:29.543013 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:29.543033 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.552539 (+ 9506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.552543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:29.554178 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:29.554194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:29.555008 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:29.555012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:29.555244 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:29.556026 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:29.556049 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:29.557839 (+ 1790us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:29.560072 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:29.560079 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:29.560100 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:29.560473 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:29.560836 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:29.560839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:29.560841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:29.561042 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:29.561521 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:29.561529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:29.561909 (+ 380us) client_negotiation.cc:769] Sending connection context
0620 06:04:29.562781 (+ 872us) client_negotiation.cc:240] Negotiation successful
0620 06:04:29.563182 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8313,"thread_start_us":175,"threads_started":1}
I20260620 06:04:29.565730 16273 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45875
I20260620 06:04:29.566035 16273 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:29.566653 16273 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:29.567723 15789 ts_manager.cc:194] Registered new tserver with Master: 0986b21987b9480595f48e6685e7ad6c (127.3.115.3:35097)
I20260620 06:04:29.568454 15789 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:51981
I20260620 06:04:29.575971 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:04:29.576819 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 15870
I20260620 06:04:29.585577 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16009
I20260620 06:04:29.592295 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16144
I20260620 06:04:29.599402 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 15757
I20260620 06:04:29.609892 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45875
--webserver_interface=127.3.115.62
--webserver_port=46217
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:45875
--ranger_config_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:29.733405 16276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:29.733736 16276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:29.733778 16276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:29.738093 16276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:29.738183 16276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:29.738204 16276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:29.738221 16276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:29.738237 16276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:29.744812 16276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:45875
--ranger_config_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=46217
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:29.746626 16276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:29.748044 16276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:29.757650 16282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:29.757825 16276 server_base.cc:1034] running on GCE node
W20260620 06:04:29.757660 16284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:29.757676 16281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:29.758626 16276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:29.759914 16276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:29.761129 16276 hybrid_clock.cc:648] HybridClock initialized: now 1781935469761100 us; error 47 us; skew 500 ppm
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:29.764321 16276 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:29.765470 16276 webserver.cc:460] Webserver started at http://127.3.115.62:46217/ using document root <none> and password file <none>
I20260620 06:04:29.766148 16276 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:29.766251 16276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:04:29 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935469, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:29.820761 16276 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:04:29.823971 16292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:29.825904 16276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:29.826085 16276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "43cc8d1e440b42bbaf57fc033f83075f"
format_stamp: "Formatted at 2026-06-20 06:04:27 on dist-test-slave-4h14"
server_key: "ab7ee0847e4983668a4ff2a186f9d5dc"
server_key_iv: "7702affef7da5e3a9b7a20e9bf51b0d0"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:29.826900 16276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:29.844197 16276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:29.847627 16276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:29.847820 16276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:29.855612 16344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:45875 every 8 connection(s)
I20260620 06:04:29.855608 16276 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:45875
I20260620 06:04:29.856920 16276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:29.860002 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16276
I20260620 06:04:29.860519 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:39491
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:45875
--webserver_port=34851
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:29.862131 16345 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:04:29.863920 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Bootstrap starting.
I20260620 06:04:29.881155 16345 log.cc:824] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:29.888028 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:29.888489 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Bootstrap complete.
I20260620 06:04:29.893011 16345 raft_consensus.cc:348] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:29.893448 16345 raft_consensus.cc:729] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cc8d1e440b42bbaf57fc033f83075f, State: Initialized, Role: FOLLOWER
I20260620 06:04:29.894006 16345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:29.894129 16345 raft_consensus.cc:388] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:29.894218 16345 raft_consensus.cc:482] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:29.894335 16345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:04:29.896443 16345 raft_consensus.cc:504] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:29.896835 16345 leader_election.cc:302] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 43cc8d1e440b42bbaf57fc033f83075f; no voters:
I20260620 06:04:29.897200 16345 leader_election.cc:288] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:04:29.897296 16349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:04:29.897500 16349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [term 2 LEADER]: Becoming Leader. State: Replica: 43cc8d1e440b42bbaf57fc033f83075f, State: Running, Role: LEADER
I20260620 06:04:29.897820 16349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } }
I20260620 06:04:29.898530 16345 sys_catalog.cc:564] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:29.898823 16349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 43cc8d1e440b42bbaf57fc033f83075f. Latest consensus state: current_term: 2 leader_uuid: "43cc8d1e440b42bbaf57fc033f83075f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } } }
I20260620 06:04:29.898943 16349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:29.899313 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "43cc8d1e440b42bbaf57fc033f83075f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "43cc8d1e440b42bbaf57fc033f83075f" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45875 } } }
I20260620 06:04:29.899518 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:29.899322 16353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:29.903079 16353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:29.903643 16345 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 06:04:29.903892 16353 catalog_manager.cc:1259] Loaded cluster ID: 543038680c814ed2a01e5de00ecdc8f9
I20260620 06:04:29.903990 16353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:29.906445 16353 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:29.907290 16353 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Loaded TSK: 0
I20260620 06:04:29.908910 16353 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:04:29.991274 16347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:29.991624 16347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:29.991712 16347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:29.996918 16347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:29.997018 16347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:29.997128 16347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:04:30.004283 16347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:39491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=34851
--tserver_master_addrs=127.3.115.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:30.006126 16347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:30.007403 16347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:30.021220 16383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:30.021229 16382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:30.021725 16347 server_base.cc:1034] running on GCE node
W20260620 06:04:30.021957 16385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:30.022450 16347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:30.023320 16347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:30.024511 16347 hybrid_clock.cc:648] HybridClock initialized: now 1781935470024447 us; error 75 us; skew 500 ppm
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:30.027979 16347 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:04:30.029402 16347 webserver.cc:460] Webserver started at http://127.3.115.1:34851/ using document root <none> and password file <none>
I20260620 06:04:30.030054 16347 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:30.030172 16347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:30.055172 16347 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.000s
I20260620 06:04:30.058790 16393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.060978 16347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:04:30.061213 16347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "68ccb4ff431a463d971437f998efe0a6"
format_stamp: "Formatted at 2026-06-20 06:04:28 on dist-test-slave-4h14"
server_key: "f53666aaaf68865aceaf76866001632d"
server_key_iv: "aaae51b2e4ea939d16c6f96a3c473eb4"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:30.061975 16347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:30.084216 16347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:30.088188 16347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:30.088426 16347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:30.089265 16347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:30.090265 16347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:30.090340 16347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.090408 16347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:30.090432 16347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.102717 16347 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:39491
I20260620 06:04:30.102733 16506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:39491 every 8 connection(s)
I20260620 06:04:30.103881 16347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:04:30.105907 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16347
I20260620 06:04:30.106396 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:42487
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:45875
--webserver_port=42027
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:30.133805 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.108265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:54511 (local address 127.3.115.62:45875)
0620 06:04:30.109010 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:30.109037 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:30.112785 (+ 3748us) server_negotiation.cc:407] Connection header received
0620 06:04:30.113159 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:30.113179 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:30.113737 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:30.114173 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:30.116431 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.118464 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.120751 (+ 2287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.121645 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.121820 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.125573 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:30.125583 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:30.125606 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:30.125647 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:30.127927 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.128806 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.128818 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.128829 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.129011 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.129547 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.129551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.129553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.130082 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:30.130204 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:30.132259 (+ 2055us) server_negotiation.cc:299] Negotiation successful
0620 06:04:30.132594 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":122,"threads_started":1}
I20260620 06:04:30.134711 16509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.106652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45875 (local address 127.3.115.1:54511)
0620 06:04:30.110867 (+ 4215us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:30.110939 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:30.112326 (+ 1387us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:30.114847 (+ 2521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:30.114861 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:30.115318 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:30.116159 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:30.116186 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.118707 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.118713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:30.120514 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:30.120537 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.122190 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.122195 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:30.122427 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.123231 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:30.123255 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:30.125249 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:30.128119 (+ 2870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:30.128127 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:30.128146 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:30.128566 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:30.129149 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:30.129152 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:30.129154 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:30.129432 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:30.130268 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:30.130276 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:30.130617 (+ 341us) client_negotiation.cc:769] Sending connection context
0620 06:04:30.130828 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:04:30.131207 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3937,"thread_start_us":147,"threads_started":1}
I20260620 06:04:30.137640 16507 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45875
I20260620 06:04:30.137917 16507 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.138527 16507 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
W20260620 06:04:30.140904 16507 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:45875 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:04:30.141040 16507 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.141243 16507 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:30.142086 16507 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.142261 16507 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
W20260620 06:04:30.143091 16507 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:04:30.258961 16510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:30.259384 16510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:30.259452 16510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:30.264598 16510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:30.264724 16510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:30.264835 16510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:04:30.271104 16510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=42027
--tserver_master_addrs=127.3.115.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:30.272501 16510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:30.273655 16510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:30.284771 16519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:30.285254 16516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:30.285495 16510 server_base.cc:1034] running on GCE node
W20260620 06:04:30.286046 16517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:30.286576 16510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:30.287436 16510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:30.288641 16510 hybrid_clock.cc:648] HybridClock initialized: now 1781935470288608 us; error 48 us; skew 500 ppm
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:30.292311 16510 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:04:30.293607 16510 webserver.cc:460] Webserver started at http://127.3.115.2:42027/ using document root <none> and password file <none>
I20260620 06:04:30.294265 16510 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:30.294411 16510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:30.313874 16510 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:04:30.317519 16527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.319428 16510 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:04:30.319630 16510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "ed289f750456487ab34f436dd43bcdba"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "c2e53fcc09b79c6b17723a15bfd6bf3f"
server_key_iv: "e37b61dadcbda801dbba9980232b7ce9"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:30.320462 16510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:30.338282 16510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:30.341928 16510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:30.342178 16510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:30.342985 16510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:30.344192 16510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:30.344254 16510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.344311 16510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:30.344331 16510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.355878 16510 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:42487
I20260620 06:04:30.355953 16640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:42487 every 8 connection(s)
I20260620 06:04:30.357339 16510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:04:30.361039 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16510
I20260620 06:04:30.361519 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.115.20:45231/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:35097
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:45875
--webserver_port=39835
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:30.380105 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.360028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:60449 (local address 127.3.115.62:45875)
0620 06:04:30.360189 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:30.360194 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:30.364281 (+ 4087us) server_negotiation.cc:407] Connection header received
0620 06:04:30.365999 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:30.366005 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:30.366083 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:30.366163 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:30.368380 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.369357 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.371509 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.371950 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.372059 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.375450 (+ 3391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:30.375454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:30.375458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:30.375493 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:30.377386 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.378090 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.378095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.378097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.378213 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.378661 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.378664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.378666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.378905 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:30.378979 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:30.379767 (+ 788us) server_negotiation.cc:299] Negotiation successful
0620 06:04:30.379922 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:04:30.382273 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.360629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45875 (local address 127.3.115.2:60449)
0620 06:04:30.364049 (+ 3420us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:30.364126 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:30.365662 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:30.366718 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:30.366733 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:30.367178 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:30.368129 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:30.368167 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.369566 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.369572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:30.371328 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:30.371343 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.372343 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.372348 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:30.372620 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.373452 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:30.373481 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:30.375209 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:30.377538 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:30.377545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:30.377567 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:30.377960 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:30.378331 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:30.378334 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:30.378336 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:30.378542 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:30.379063 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:30.379070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:30.379471 (+ 401us) client_negotiation.cc:769] Sending connection context
0620 06:04:30.380620 (+ 1149us) client_negotiation.cc:240] Negotiation successful
0620 06:04:30.381058 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3047,"thread_start_us":180,"threads_started":1}
I20260620 06:04:30.384050 16641 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45875
I20260620 06:04:30.384316 16641 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.384946 16641 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
W20260620 06:04:30.386902 16641 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:45875 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:04:30.387050 16641 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.387246 16641 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:30.388180 16641 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.388348 16641 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
W20260620 06:04:30.389164 16641 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:04:30.525866 16644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:30.526278 16644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:30.526363 16644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:30.531253 16644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:30.531360 16644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:30.531450 16644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:04:30.537752 16644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.115.20:45231/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:35097
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=39835
--tserver_master_addrs=127.3.115.62:45875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16644
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:30.539067 16644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:30.540207 16644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:30.551579 16650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:30.551585 16653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:30.551640 16651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:30.552304 16644 server_base.cc:1034] running on GCE node
I20260620 06:04:30.552883 16644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:30.553781 16644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:30.554996 16644 hybrid_clock.cc:648] HybridClock initialized: now 1781935470554972 us; error 35 us; skew 500 ppm
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:30.558784 16644 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:04:30.560025 16644 webserver.cc:460] Webserver started at http://127.3.115.3:39835/ using document root <none> and password file <none>
I20260620 06:04:30.560839 16644 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:30.560961 16644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:30.581159 16644 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:04:30.591921 16661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.594092 16644 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.003s sys 0.000s
I20260620 06:04:30.594331 16644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "0986b21987b9480595f48e6685e7ad6c"
format_stamp: "Formatted at 2026-06-20 06:04:29 on dist-test-slave-4h14"
server_key: "6f630621630562b448ed5d7d3ea4660d"
server_key_iv: "a8e2b7d7b92e260641cebaebf4791207"
server_key_version: "kuduclusterkey@0"
I20260620 06:04:30.595196 16644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:30.622215 16644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:30.626255 16644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:30.626475 16644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:30.627178 16644 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:30.628113 16644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:30.628177 16644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.628255 16644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:30.628290 16644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:30.639735 16644 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:35097
I20260620 06:04:30.639755 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:35097 every 8 connection(s)
I20260620 06:04:30.641145 16644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
Jun 20 06:04:30 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935470, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:30.651156 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16644
I20260620 06:04:30.665104 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.643158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:55889 (local address 127.3.115.62:45875)
0620 06:04:30.643315 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:30.643320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:30.644389 (+ 1069us) server_negotiation.cc:407] Connection header received
0620 06:04:30.645460 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:30.645464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:30.645520 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:30.645612 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:30.646854 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.647389 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.648709 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.648958 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.649069 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.651494 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:30.651497 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:30.651500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:30.651535 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:30.653346 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.655221 (+ 1875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.655225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.655227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.655329 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.655853 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.655858 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.655861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.656115 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:30.656203 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:30.664739 (+ 8536us) server_negotiation.cc:299] Negotiation successful
0620 06:04:30.664894 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:04:30.667430 16777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.643689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45875 (local address 127.3.115.3:55889)
0620 06:04:30.644222 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:30.644279 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:30.645238 (+ 959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:30.645831 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:30.645842 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:30.646183 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:30.646704 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:30.646718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.647517 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.647521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:30.648564 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:30.648574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.649114 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.649118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:30.649298 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.649919 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:30.649936 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:30.651325 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:30.653855 (+ 2530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:30.653864 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:30.653885 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:30.654483 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:30.655477 (+ 994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:30.655480 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:30.655482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:30.655731 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:30.663935 (+ 8204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:30.663948 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:30.664381 (+ 433us) client_negotiation.cc:769] Sending connection context
0620 06:04:30.664628 (+ 247us) client_negotiation.cc:240] Negotiation successful
0620 06:04:30.666018 (+ 1390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260620 06:04:30.669003 16775 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45875
I20260620 06:04:30.669296 16775 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.669880 16779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:30.652301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38614 (local address 127.3.115.62:45875)
0620 06:04:30.652604 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:30.652612 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:30.652672 (+ 60us) server_negotiation.cc:407] Connection header received
0620 06:04:30.653658 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:30.653662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:30.653718 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:30.653819 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:30.654797 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.655572 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.657854 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:30.658217 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:30.658335 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:30.663404 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:30.663409 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:30.663412 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:30.663450 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:30.666560 (+ 3110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.667177 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.667181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.667183 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.667351 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:30.668063 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:30.668068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:30.668070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:30.668340 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:30.668557 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:30.669258 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 06:04:30.669673 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":105,"threads_started":1}
I20260620 06:04:30.669945 16775 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
W20260620 06:04:30.672271 16775 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:45875 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 06:04:30.672430 16775 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.672693 16775 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:30.673602 16775 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:30.673806 16775 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
W20260620 06:04:30.674618 16775 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 06:04:31 dist-test-slave-4h14 krb5kdc[15313](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781935471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:31.143337 16507 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:31.143656 16507 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:31.144882 16309 ts_manager.cc:194] Registered new tserver with Master: 68ccb4ff431a463d971437f998efe0a6 (127.3.115.1:39491)
I20260620 06:04:31.146648 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:54511
I20260620 06:04:31.389394 16641 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:31.389751 16641 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:31.390576 16309 ts_manager.cc:194] Registered new tserver with Master: ed289f750456487ab34f436dd43bcdba (127.3.115.2:42487)
I20260620 06:04:31.391407 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:60449
Jun 20 06:04:31 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:31.674856 16775 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:31.675163 16775 heartbeater.cc:511] Master 127.3.115.62:45875 requested a full tablet report, sending...
I20260620 06:04:31.675954 16309 ts_manager.cc:194] Registered new tserver with Master: 0986b21987b9480595f48e6685e7ad6c (127.3.115.3:35097)
I20260620 06:04:31.676784 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:55889
I20260620 06:04:31.686460 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 06:04:31 dist-test-slave-4h14 krb5kdc[15313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935471, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for HTTP/127.3.115.20@KRBTEST.COM
I20260620 06:04:31.709826 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.688139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52044 (local address 127.3.115.62:45875)
0620 06:04:31.688768 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.688773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.689490 (+ 717us) server_negotiation.cc:407] Connection header received
0620 06:04:31.689617 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.689620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.689676 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.689790 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:31.691632 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.694377 (+ 2745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.695645 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.696032 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.696147 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.698618 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:31.698622 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:31.698625 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:31.698660 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:31.700537 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:31.701364 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:31.701368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:31.701370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:31.701609 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:31.702005 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:31.702009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:31.702010 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:31.702257 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:31.708152 (+ 5895us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.709434 (+ 1282us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.709602 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":100,"threads_started":1}
I20260620 06:04:31.714922 16309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:04:31.720433 16309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:04:31.749606 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.741106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59846 (local address 127.3.115.1:39491)
0620 06:04:31.741552 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.741559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.743539 (+ 1980us) server_negotiation.cc:407] Connection header received
0620 06:04:31.743648 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.743654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.743931 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.744124 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:31.744710 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.745954 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.747629 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.748513 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.748595 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.748813 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.749292 (+ 479us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.749414 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":106,"threads_started":1}
I20260620 06:04:31.750471 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.740924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:39491 (local address 127.0.0.1:59846)
0620 06:04:31.743429 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.743442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:31.743525 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:31.744329 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:31.744336 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:31.744362 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:31.744509 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.744517 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.746104 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.746107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.747475 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.747486 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.748919 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.748923 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.749045 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.749101 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 06:04:31.750108 (+ 1007us) client_negotiation.cc:240] Negotiation successful
0620 06:04:31.750296 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2445,"mutex_wait_us":9,"thread_start_us":115,"threads_started":1}
I20260620 06:04:31.754029 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.738968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:35097 (local address 127.0.0.1:46898)
0620 06:04:31.742999 (+ 4031us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.743066 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:31.743204 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:31.745486 (+ 2282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:31.745489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:31.745506 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:31.745608 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.745614 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.747936 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.747940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.750305 (+ 2365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.750316 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.753587 (+ 3271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.753591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.753703 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.753728 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:04:31.753795 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 06:04:31.753872 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3832,"mutex_wait_us":1,"thread_start_us":114,"threads_started":1}
I20260620 06:04:31.754984 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.739938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.3.115.3:35097)
0620 06:04:31.743797 (+ 3859us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.743804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.743831 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:04:31.743912 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.743918 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.745159 (+ 1241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.745314 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:31.745753 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.747795 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.750451 (+ 2656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.752499 (+ 2048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.752616 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.754669 (+ 2053us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.754740 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.754843 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3722,"thread_start_us":140,"threads_started":1}
I20260620 06:04:31.754477 16440 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3984ef3a79469f82eaf1a84f6a5b39 (DEFAULT_TABLE table=test-table [id=e0ae951eebe34ab3a3d50f6e9ad67ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:31.755748 16440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3984ef3a79469f82eaf1a84f6a5b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:31.756125 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.741654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42487 (local address 127.0.0.1:53866)
0620 06:04:31.750895 (+ 9241us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.750911 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:31.750994 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:31.751900 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:31.751903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:31.751923 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:31.752002 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.752008 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.753412 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.753416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.754499 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.754512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.755735 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.755738 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.755837 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.755861 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:04:31.755934 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:04:31.756005 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9162,"thread_start_us":97,"threads_started":1}
I20260620 06:04:31.756626 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.741827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.3.115.2:42487)
0620 06:04:31.742204 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.742211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.751113 (+ 8902us) server_negotiation.cc:407] Connection header received
0620 06:04:31.751219 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.751228 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.751510 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.751688 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:31.752131 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.753270 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.754647 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.755430 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.755529 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.755658 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.756389 (+ 731us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.756493 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":103,"threads_started":1}
I20260620 06:04:31.758361 16703 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3984ef3a79469f82eaf1a84f6a5b39 (DEFAULT_TABLE table=test-table [id=e0ae951eebe34ab3a3d50f6e9ad67ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:31.760437 16703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3984ef3a79469f82eaf1a84f6a5b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:31.762084 16563 tablet_service.cc:1467] Processing CreateTablet for tablet 2d3984ef3a79469f82eaf1a84f6a5b39 (DEFAULT_TABLE table=test-table [id=e0ae951eebe34ab3a3d50f6e9ad67ee1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:31.763228 16563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d3984ef3a79469f82eaf1a84f6a5b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:31.765829 16809 tablet_bootstrap.cc:492] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: Bootstrap starting.
I20260620 06:04:31.770782 16809 tablet_bootstrap.cc:654] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:31.771674 16811 tablet_bootstrap.cc:492] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: Bootstrap starting.
I20260620 06:04:31.771833 16809 log.cc:824] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:31.772593 16812 tablet_bootstrap.cc:492] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Bootstrap starting.
I20260620 06:04:31.774923 16811 tablet_bootstrap.cc:654] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:31.774922 16812 tablet_bootstrap.cc:654] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:31.775815 16812 log.cc:824] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:31.775863 16811 log.cc:824] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:31.778280 16811 tablet_bootstrap.cc:492] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: No bootstrap required, opened a new log
I20260620 06:04:31.778554 16811 ts_tablet_manager.cc:1397] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:04:31.778754 16812 tablet_bootstrap.cc:492] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: No bootstrap required, opened a new log
I20260620 06:04:31.778940 16812 ts_tablet_manager.cc:1397] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:04:31.782857 16809 tablet_bootstrap.cc:492] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: No bootstrap required, opened a new log
I20260620 06:04:31.783100 16809 ts_tablet_manager.cc:1397] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20260620 06:04:31.783300 16812 raft_consensus.cc:348] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.783646 16812 raft_consensus.cc:374] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:31.783731 16812 raft_consensus.cc:729] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0986b21987b9480595f48e6685e7ad6c, State: Initialized, Role: FOLLOWER
I20260620 06:04:31.784267 16812 consensus_queue.cc:260] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.784561 16811 raft_consensus.cc:348] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.785064 16811 raft_consensus.cc:374] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:31.785172 16811 raft_consensus.cc:729] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed289f750456487ab34f436dd43bcdba, State: Initialized, Role: FOLLOWER
I20260620 06:04:31.785485 16812 ts_tablet_manager.cc:1428] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Time spent starting tablet: real 0.006s user 0.002s sys 0.006s
I20260620 06:04:31.785795 16811 consensus_queue.cc:260] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.785928 16775 heartbeater.cc:503] Master 127.3.115.62:45875 was elected leader, sending a full tablet report...
I20260620 06:04:31.786938 16811 ts_tablet_manager.cc:1428] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260620 06:04:31.787279 16641 heartbeater.cc:503] Master 127.3.115.62:45875 was elected leader, sending a full tablet report...
I20260620 06:04:31.789166 16809 raft_consensus.cc:348] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.790434 16809 raft_consensus.cc:374] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:31.790542 16809 raft_consensus.cc:729] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ccb4ff431a463d971437f998efe0a6, State: Initialized, Role: FOLLOWER
I20260620 06:04:31.792006 16809 consensus_queue.cc:260] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.793622 16507 heartbeater.cc:503] Master 127.3.115.62:45875 was elected leader, sending a full tablet report...
I20260620 06:04:31.793992 16817 raft_consensus.cc:482] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:31.794203 16817 raft_consensus.cc:504] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.794273 16809 ts_tablet_manager.cc:1428] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260620 06:04:31.795449 16817 leader_election.cc:288] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0986b21987b9480595f48e6685e7ad6c (127.3.115.3:35097), 68ccb4ff431a463d971437f998efe0a6 (127.3.115.1:39491)
I20260620 06:04:31.801015 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.796116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:45099 (local address 127.3.115.1:39491)
0620 06:04:31.796226 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.796250 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.796477 (+ 227us) server_negotiation.cc:407] Connection header received
0620 06:04:31.796841 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.796844 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.796895 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.796945 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:31.797417 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.797948 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.799613 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.800242 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.800316 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.800378 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.800785 (+ 407us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.800886 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:04:31.801669 16820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.795712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:39491 (local address 127.3.115.2:45099)
0620 06:04:31.796390 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.796405 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:31.796644 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:31.797099 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:31.797102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:31.797122 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:31.797290 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.797299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.798126 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.798129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.799452 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.799461 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.800490 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.800493 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.800618 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.800644 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:04:31.801316 (+ 672us) client_negotiation.cc:240] Negotiation successful
0620 06:04:31.801500 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":124,"threads_started":1}
I20260620 06:04:31.802561 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3984ef3a79469f82eaf1a84f6a5b39" candidate_uuid: "ed289f750456487ab34f436dd43bcdba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ccb4ff431a463d971437f998efe0a6" is_pre_election: true
I20260620 06:04:31.802943 16459 raft_consensus.cc:2454] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed289f750456487ab34f436dd43bcdba in term 0.
I20260620 06:04:31.803547 16819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.796095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:35097 (local address 127.3.115.2:60703)
0620 06:04:31.799054 (+ 2959us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.799070 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:31.799191 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:31.799523 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:31.799529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:31.799552 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:31.799670 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.799676 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.800644 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.800647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.801936 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:31.802008 (+ 72us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.803088 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.803092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:31.803218 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.803253 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:04:31.803332 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 06:04:31.803402 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2890,"thread_start_us":114,"threads_started":1}
I20260620 06:04:31.803568 16804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.795890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:60703 (local address 127.3.115.3:35097)
0620 06:04:31.796006 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.796011 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.799190 (+ 3179us) server_negotiation.cc:407] Connection header received
0620 06:04:31.799295 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.799297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.799349 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.799421 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:31.799793 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.800506 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.802280 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.802744 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.802852 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.802916 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.803367 (+ 451us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.803452 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:04:31.803476 16530 leader_election.cc:302] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ccb4ff431a463d971437f998efe0a6, ed289f750456487ab34f436dd43bcdba; no voters:
I20260620 06:04:31.803939 16817 raft_consensus.cc:2781] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:04:31.804049 16817 raft_consensus.cc:482] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:04:31.804100 16817 raft_consensus.cc:3037] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:31.804502 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3984ef3a79469f82eaf1a84f6a5b39" candidate_uuid: "ed289f750456487ab34f436dd43bcdba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0986b21987b9480595f48e6685e7ad6c" is_pre_election: true
I20260620 06:04:31.804847 16717 raft_consensus.cc:2454] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed289f750456487ab34f436dd43bcdba in term 0.
I20260620 06:04:31.805500 16817 raft_consensus.cc:504] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.805886 16817 leader_election.cc:288] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [CANDIDATE]: Term 1 election: Requested vote from peers 0986b21987b9480595f48e6685e7ad6c (127.3.115.3:35097), 68ccb4ff431a463d971437f998efe0a6 (127.3.115.1:39491)
I20260620 06:04:31.806211 16717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3984ef3a79469f82eaf1a84f6a5b39" candidate_uuid: "ed289f750456487ab34f436dd43bcdba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0986b21987b9480595f48e6685e7ad6c"
I20260620 06:04:31.806344 16717 raft_consensus.cc:3037] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:31.806371 16459 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d3984ef3a79469f82eaf1a84f6a5b39" candidate_uuid: "ed289f750456487ab34f436dd43bcdba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ccb4ff431a463d971437f998efe0a6"
I20260620 06:04:31.806481 16459 raft_consensus.cc:3037] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:31.807724 16717 raft_consensus.cc:2454] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed289f750456487ab34f436dd43bcdba in term 1.
I20260620 06:04:31.807724 16459 raft_consensus.cc:2454] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed289f750456487ab34f436dd43bcdba in term 1.
I20260620 06:04:31.808055 16528 leader_election.cc:302] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0986b21987b9480595f48e6685e7ad6c, ed289f750456487ab34f436dd43bcdba; no voters:
I20260620 06:04:31.808238 16817 raft_consensus.cc:2781] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:31.808470 16817 raft_consensus.cc:686] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 1 LEADER]: Becoming Leader. State: Replica: ed289f750456487ab34f436dd43bcdba, State: Running, Role: LEADER
I20260620 06:04:31.808902 16817 consensus_queue.cc:237] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } }
I20260620 06:04:31.812206 16308 catalog_manager.cc:5495] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba reported cstate change: term changed from 0 to 1, leader changed from <none> to ed289f750456487ab34f436dd43bcdba (127.3.115.2). New cstate: current_term: 1 leader_uuid: "ed289f750456487ab34f436dd43bcdba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ed289f750456487ab34f436dd43bcdba" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42487 } health_report { overall_health: HEALTHY } } }
I20260620 06:04:31.838517 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.833318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.3.115.2:42487)
0620 06:04:31.833437 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:31.833442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:31.833456 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:04:31.833548 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:31.833551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:31.833618 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:31.833798 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:04:31.834109 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.834871 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.836103 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:31.836452 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:31.836563 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:31.837062 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:04:31.837702 (+ 640us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:04:31.837832 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:31.838279 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 06:04:31.838345 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:04:31.846257 16459 raft_consensus.cc:1261] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 1 FOLLOWER]: Refusing update from remote peer ed289f750456487ab34f436dd43bcdba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:31.846386 16717 raft_consensus.cc:1261] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 1 FOLLOWER]: Refusing update from remote peer ed289f750456487ab34f436dd43bcdba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:31.847055 16817 consensus_queue.cc:1035] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [LEADER]: Connected to new peer: Peer: permanent_uuid: "0986b21987b9480595f48e6685e7ad6c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 35097 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:31.847240 16821 consensus_queue.cc:1035] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ccb4ff431a463d971437f998efe0a6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 39491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:31.860445 16824 mvcc.cc:205] Tried to move back new op lower bound from 7298807692673925120 to 7298807692532125696. Current Snapshot: MvccSnapshot[applied={T|T < 7298807692673925120}]
I20260620 06:04:31.863487 16826 mvcc.cc:205] Tried to move back new op lower bound from 7298807692673925120 to 7298807692532125696. Current Snapshot: MvccSnapshot[applied={T|T < 7298807692673925120}]
I20260620 06:04:31.873332 16308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:31.873535 16308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:31.876446 16308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 43cc8d1e440b42bbaf57fc033f83075f: Sending DeleteTablet for 3 replicas of tablet 2d3984ef3a79469f82eaf1a84f6a5b39
I20260620 06:04:31.877825 16703 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3984ef3a79469f82eaf1a84f6a5b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:31 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:46898
I20260620 06:04:31.878329 16440 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3984ef3a79469f82eaf1a84f6a5b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:31 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:59846
I20260620 06:04:31.878839 16834 tablet_replica.cc:330] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6: stopping tablet replica
I20260620 06:04:31.878871 16563 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d3984ef3a79469f82eaf1a84f6a5b39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:31 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:53866
I20260620 06:04:31.879303 16834 raft_consensus.cc:2229] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 68ccb4ff431a463d971437f998efe0a6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:04:31.879962 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16347
I20260620 06:04:31.892942 16833 tablet_replica.cc:330] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: stopping tablet replica
I20260620 06:04:31.893391 16833 raft_consensus.cc:2229] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:04:31.893800 16833 raft_consensus.cc:2258] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:31.895750 16833 ts_tablet_manager.cc:1905] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:04:31.898906 16833 ts_tablet_manager.cc:1918] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:31.899451 16833 log.cc:1196] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/2d3984ef3a79469f82eaf1a84f6a5b39
I20260620 06:04:31.899874 16835 tablet_replica.cc:330] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba: stopping tablet replica
I20260620 06:04:31.900373 16833 ts_tablet_manager.cc:1939] T 2d3984ef3a79469f82eaf1a84f6a5b39 P 0986b21987b9480595f48e6685e7ad6c: Deleting consensus metadata
I20260620 06:04:31.903733 16835 raft_consensus.cc:2229] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:04:31.903879 16293 catalog_manager.cc:4841] TS 0986b21987b9480595f48e6685e7ad6c (127.3.115.3:35097): tablet 2d3984ef3a79469f82eaf1a84f6a5b39 (table test-table [id=e0ae951eebe34ab3a3d50f6e9ad67ee1]) successfully deleted
I20260620 06:04:31.904378 16835 raft_consensus.cc:2258] T 2d3984ef3a79469f82eaf1a84f6a5b39 P ed289f750456487ab34f436dd43bcdba [term 1 FOLLOWER]: Raft consensus is shut down!
W20260620 06:04:31.908814 16295 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:39491 (error 108)
I20260620 06:04:31.910408 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16510
I20260620 06:04:31.911916 16803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:31.911576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:39491 (local address 127.0.0.1:59852)
0620 06:04:31.911722 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.911816 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:39491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:04:31.912189 16295 catalog_manager.cc:4584] TS 68ccb4ff431a463d971437f998efe0a6 (127.3.115.1:39491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d3984ef3a79469f82eaf1a84f6a5b39: Network error: Client connection negotiation failed: client connection to 127.3.115.1:39491: connect: Connection refused (error 111)
W20260620 06:04:31.920233 16803 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:04:31.919859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42487 (local address 127.0.0.1:53884)
0620 06:04:31.920025 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:31.920094 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.2:42487: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 06:04:31.920403 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16644
I20260620 06:04:31.931474 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16276
I20260620 06:04:31.942090 3532 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 06:04:32.277442 3532 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 06:04:32.277557 3532 mini_ranger.cc:66] Stopping Ranger...
I20260620 06:04:32.623678 3532 mini_ranger.cc:68] Stopped Ranger
2026-06-20 06:04:32.623 UTC [15343] LOG: received smart shutdown request
2026-06-20 06:04:32.628 UTC [15343] LOG: background worker "logical replication launcher" (PID 15352) exited with exit code 1
2026-06-20 06:04:32.628 UTC [15347] LOG: shutting down
2026-06-20 06:04:32.834 UTC [15343] LOG: database system is shut down
2026-06-20T06:04:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50242 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:04:33 dist-test-slave-4h14 krb5kdc[16845](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:04:33 dist-test-slave-4h14 krb5kdc[16845](info): set up 2 sockets
Jun 20 06:04:33 dist-test-slave-4h14 krb5kdc[16845](info): commencing operation
krb5kdc: starting...
W20260620 06:04:34.024967 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:04:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:04:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:04:34.161576 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:36199
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:36199,127.3.115.61:36525,127.3.115.60:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:34.282459 16858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:34.282801 16858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:34.282846 16858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:34.287293 16858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:34.287395 16858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:34.287415 16858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:34.287433 16858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:34.287449 16858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:34.293671 16858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:36199,127.3.115.61:36525,127.3.115.60:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:36199
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:34.294847 16858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:34.295917 16858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:34.305342 16863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:34.305413 16864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:34.305416 16866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:34.305548 16858 server_base.cc:1034] running on GCE node
I20260620 06:04:34.306413 16858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:34.307590 16858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:34.308737 16858 hybrid_clock.cc:648] HybridClock initialized: now 1781935474308710 us; error 51 us; skew 500 ppm
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:34.311826 16858 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:34.312875 16858 webserver.cc:460] Webserver started at http://127.3.115.62:36385/ using document root <none> and password file <none>
I20260620 06:04:34.313462 16858 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:34.313536 16858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:34.313717 16858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:34.315918 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "88b6dedd9a534266b0770e48e2f4993b"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "4cbd1f0e985156293486442900b161a3"
server_key_iv: "f19e100b2011b1ad5910fac60478846a"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.316465 16858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "88b6dedd9a534266b0770e48e2f4993b"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "4cbd1f0e985156293486442900b161a3"
server_key_iv: "f19e100b2011b1ad5910fac60478846a"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.320489 16858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:04:34.323307 16873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:34.325093 16858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:34.325269 16858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "88b6dedd9a534266b0770e48e2f4993b"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "4cbd1f0e985156293486442900b161a3"
server_key_iv: "f19e100b2011b1ad5910fac60478846a"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.325402 16858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:34.348371 16858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:34.352105 16858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:34.352335 16858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:34.360402 16858 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:36199
I20260620 06:04:34.360513 16925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:36199 every 8 connection(s)
I20260620 06:04:34.361491 16858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:34.364450 16926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:34.368428 16926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.369230 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16858
I20260620 06:04:34.369377 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:04:34.369669 3532 external_mini_cluster.cc:1428] Setting key 66973524b27b7c031eac6e032a9b4b89
WARNING: no policy specified for kudu/127.3.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.61@KRBTEST.COM" created.
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:34.387609 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.371103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36199 (local address 127.0.0.1:49570)
0620 06:04:34.371757 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.371815 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.372846 (+ 1031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.373667 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.373676 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.374220 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.374669 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.374694 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.375924 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.375929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.377296 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.377309 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.378075 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.378079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.378197 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.378791 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.378809 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.381975 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.384257 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.384270 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.384286 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.384632 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.385009 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.385015 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.385020 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.385190 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.385912 (+ 722us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.385923 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.386144 (+ 221us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.386360 (+ 216us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.386798 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":55,"spinlock_wait_cycles":57600,"thread_start_us":199,"threads_started":1}
I20260620 06:04:34.387612 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.371102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.3.115.62:36199)
0620 06:04:34.372214 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.372227 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.372271 (+ 44us) server_negotiation.cc:407] Connection header received
0620 06:04:34.373184 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.373189 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.373383 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.373603 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.374878 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.375791 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.377446 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.377890 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.378080 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.382159 (+ 4079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.382170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.382189 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.382224 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.384134 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.384794 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.384800 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.384801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.384912 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.385297 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.385300 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.385301 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.385787 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.385895 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.386433 (+ 538us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.386720 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":990,"thread_start_us":174,"threads_started":1}
I20260620 06:04:34.390314 16926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.391110 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.390853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41630)
0620 06:04:34.390973 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.391044 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:04:34.391348 16876 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
I20260620 06:04:34.392275 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.392070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41642)
0620 06:04:34.392174 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.392222 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:04:34.392417 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:36525: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab.
Entry for principal kudu/127.3.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab.
I20260620 06:04:34.400336 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:36525
--webserver_interface=127.3.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab
--principal=kudu/127.3.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:36199,127.3.115.61:36525,127.3.115.60:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:34.441694 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } attempt: 1
I20260620 06:04:34.442718 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.442413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41658)
0620 06:04:34.442570 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.442635 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 06:04:34.443451 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.443231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41664)
0620 06:04:34.443336 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.443391 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:04:34.443588 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:36525: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
I20260620 06:04:34.511797 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } attempt: 2
I20260620 06:04:34.512794 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.512426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41678)
0620 06:04:34.512588 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.512705 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 06:04:34.513556 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.513337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41686)
0620 06:04:34.513445 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.513502 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:04:34.513669 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:36525: Network error: Client connection negotiation failed: client connection to 127.3.115.61:36525: connect: Connection refused (error 111)
W20260620 06:04:34.520278 16933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:34.520614 16933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:34.520725 16933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:34.525100 16933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:34.525185 16933 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:34.525207 16933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:34.525224 16933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:34.525240 16933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:34.531172 16933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:36199,127.3.115.61:36525,127.3.115.60:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab
--principal=kudu/127.3.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:36525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16933
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:34.532374 16933 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:34.533530 16933 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:04:34.542331 16933 server_base.cc:1034] running on GCE node
W20260620 06:04:34.542367 16939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:34.542348 16938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:34.542308 16941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:34.543205 16933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:34.544401 16933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:34.545603 16933 hybrid_clock.cc:648] HybridClock initialized: now 1781935474545599 us; error 62 us; skew 500 ppm
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:34.548811 16933 init.cc:377] Logged in from keytab as kudu/127.3.115.61@KRBTEST.COM (short username kudu)
I20260620 06:04:34.549882 16933 webserver.cc:460] Webserver started at http://127.3.115.61:42715/ using document root <none> and password file <none>
I20260620 06:04:34.550504 16933 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:34.550611 16933 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:34.550845 16933 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:34.553246 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data/instance:
uuid: "8d8b819e63a74efc85dab06236b0da24"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "2ec833b7adea3fa4ea64ca9b60ee1c42"
server_key_iv: "3dc6f89f12cf39601ab862ab4b94a754"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.553901 16933 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal/instance:
uuid: "8d8b819e63a74efc85dab06236b0da24"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "2ec833b7adea3fa4ea64ca9b60ee1c42"
server_key_iv: "3dc6f89f12cf39601ab862ab4b94a754"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.558256 16933 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:04:34.561110 16948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:34.562441 16933 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:04:34.562602 16933 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal
uuid: "8d8b819e63a74efc85dab06236b0da24"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "2ec833b7adea3fa4ea64ca9b60ee1c42"
server_key_iv: "3dc6f89f12cf39601ab862ab4b94a754"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.562738 16933 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:34.583314 16933 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:34.587167 16933 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:34.587376 16933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:34.595110 16933 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.61:36525
I20260620 06:04:34.595126 17000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.61:36525 every 8 connection(s)
I20260620 06:04:34.596247 16933 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
I20260620 06:04:34.597410 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 16933
I20260620 06:04:34.597556 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-1/wal/instance
I20260620 06:04:34.597841 3532 external_mini_cluster.cc:1428] Setting key 04e2199d87c0158ec04ee0b14ac43668
I20260620 06:04:34.599690 17001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:34.604838 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } attempt: 3
I20260620 06:04:34.606284 17001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.115.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.60@KRBTEST.COM" created.
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:34.620280 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.608573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.3.115.62:36199)
0620 06:04:34.608740 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.608745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.609218 (+ 473us) server_negotiation.cc:407] Connection header received
0620 06:04:34.609348 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.609350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.609394 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.609463 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.610428 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.611666 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.612884 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.613243 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.613365 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.615836 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.615840 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.615843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.615877 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.617987 (+ 2110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.618585 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.618588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.618590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.618686 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.619023 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.619028 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.619030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.619228 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.619333 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.619791 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.620019 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":55,"spinlock_wait_cycles":2304}
I20260620 06:04:34.620137 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.605411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41698)
0620 06:04:34.605657 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.605674 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.605782 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.608336 (+ 2554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.608339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.608585 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.608779 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.608791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.610647 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.610654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.611409 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.611420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.612373 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.612380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.612527 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.613164 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.613185 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.614987 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.617706 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.617712 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.617716 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.618107 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.618508 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.618511 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.618515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.618637 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.619558 (+ 921us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.619561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.619667 (+ 106us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.619771 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.619931 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"spinlock_wait_cycles":896}
I20260620 06:04:34.621312 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.608512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36199 (local address 127.0.0.1:49586)
0620 06:04:34.609030 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.609082 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.609254 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.609536 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.609542 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.610013 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.610260 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.610274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.611807 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.611810 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.612770 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.612782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.613416 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.613421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.613533 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.614073 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.614089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.615653 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.618095 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.618105 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.618146 (+ 41us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.618464 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.618785 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.618788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.618790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.618931 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.619333 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.619341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.619494 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.619724 (+ 230us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.620084 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":184,"threads_started":1}
I20260620 06:04:34.621312 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.606233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41698 (local address 127.3.115.61:36525)
0620 06:04:34.606733 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.606758 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.606798 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:04:34.607570 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.607589 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.607982 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.608361 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.608971 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.610536 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.611536 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.612235 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.612409 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.615161 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.615170 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.615188 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.615221 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.617572 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.618257 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.618265 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.618267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.618403 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.618764 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.618768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.618772 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.619432 (+ 660us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.619582 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.620310 (+ 728us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.620452 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":155,"threads_started":1}
I20260620 06:04:34.622704 17001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.623608 16926 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.624325 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.624078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36298)
0620 06:04:34.624198 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.624254 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 06:04:34.625175 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.624890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36308)
0620 06:04:34.625032 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.625086 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 06:04:34.625319 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab.
Entry for principal kudu/127.3.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab.
I20260620 06:04:34.632066 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.623439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41702 (local address 127.3.115.61:36525)
0620 06:04:34.623537 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.623548 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.623562 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:04:34.623597 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.623599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.623647 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.623719 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.624232 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.624942 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.625938 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.626163 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.626268 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.628540 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.628543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.628546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.628578 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.630299 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.630788 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.630792 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.630795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.630879 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.631177 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.631181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.631184 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.631427 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.631529 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.631756 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.631942 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":105,"server-negotiator.queue_time_us":36}
I20260620 06:04:34.632092 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:40387
--webserver_interface=127.3.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab
--principal=kudu/127.3.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:42003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:36199,127.3.115.61:36525,127.3.115.60:40387
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:34.632066 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.623270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41702)
0620 06:04:34.623392 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.623413 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.623516 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.623755 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.623763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.624029 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.624114 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.624120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.625076 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.625079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.625746 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.625756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.626265 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.626269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.626392 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.626972 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.626986 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.628349 (+ 1363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.630421 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.630426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.630429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.630666 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.630990 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.630993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.630995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.631091 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.631516 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.631519 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.631608 (+ 89us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.631711 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.631870 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":22}
I20260620 06:04:34.633379 17001 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.634243 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.633890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36316)
0620 06:04:34.634024 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.634142 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:04:34.634564 16952 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
I20260620 06:04:34.635437 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.635224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36328)
0620 06:04:34.635329 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.635376 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:04:34.635593 17001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
I20260620 06:04:34.656510 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 1
I20260620 06:04:34.657454 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.657149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36338)
0620 06:04:34.657298 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.657365 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:04:34.658258 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.658007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36354)
0620 06:04:34.658144 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.658198 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:04:34.658425 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
I20260620 06:04:34.684870 17001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 1
I20260620 06:04:34.685873 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.685493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36370)
0620 06:04:34.685685 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.685776 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 06:04:34.686735 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.686484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36374)
0620 06:04:34.686602 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.686660 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:04:34.686877 17001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
I20260620 06:04:34.733645 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 2
I20260620 06:04:34.734643 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.734301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36382)
0620 06:04:34.734485 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.734552 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260620 06:04:34.735416 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.735196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36398)
0620 06:04:34.735309 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.735361 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:04:34.735548 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
W20260620 06:04:34.753366 17007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:34.753728 17007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:34.753801 17007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 06:04:34.755085 17001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 2
I20260620 06:04:34.756017 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.755657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36408)
0620 06:04:34.755842 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.755921 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 06:04:34.756880 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.756622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36420)
0620 06:04:34.756756 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.756809 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260620 06:04:34.757017 17001 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
W20260620 06:04:34.758809 17007 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:34.758869 17007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:34.758890 17007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:34.758908 17007 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:34.758924 17007 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:34.764933 17007 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:42003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:36199,127.3.115.61:36525,127.3.115.60:40387
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab
--principal=kudu/127.3.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:40387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:34.766150 17007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:34.767365 17007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:34.777539 17014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:34.777539 17016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:34.777551 17013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:34.777810 17007 server_base.cc:1034] running on GCE node
I20260620 06:04:34.778453 17007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:34.779745 17007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:34.780958 17007 hybrid_clock.cc:648] HybridClock initialized: now 1781935474780936 us; error 39 us; skew 500 ppm
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:34.784413 17007 init.cc:377] Logged in from keytab as kudu/127.3.115.60@KRBTEST.COM (short username kudu)
I20260620 06:04:34.785526 17007 webserver.cc:460] Webserver started at http://127.3.115.60:33199/ using document root <none> and password file <none>
I20260620 06:04:34.786170 17007 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:34.786242 17007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:34.786425 17007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:34.788712 17007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data/instance:
uuid: "166ff9baf2b647ad95ce7dcd03521aff"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "1b90d91a6ca8cfadd0c00ab0a9c598c6"
server_key_iv: "35c49a8911872d32a88a1882b2d33204"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.789333 17007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal/instance:
uuid: "166ff9baf2b647ad95ce7dcd03521aff"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "1b90d91a6ca8cfadd0c00ab0a9c598c6"
server_key_iv: "35c49a8911872d32a88a1882b2d33204"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.793570 17007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:04:34.796434 17023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:34.798074 17007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:34.798218 17007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal
uuid: "166ff9baf2b647ad95ce7dcd03521aff"
format_stamp: "Formatted at 2026-06-20 06:04:34 on dist-test-slave-4h14"
server_key: "1b90d91a6ca8cfadd0c00ab0a9c598c6"
server_key_iv: "35c49a8911872d32a88a1882b2d33204"
server_key_version: "encryptionkey@0"
I20260620 06:04:34.798336 17007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:34.825835 17007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:34.829708 17007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:34.829924 17007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:34.834712 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 3
I20260620 06:04:34.835690 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.835362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36436)
0620 06:04:34.835525 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.835596 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 06:04:34.836421 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.836189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36450)
0620 06:04:34.836292 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.836363 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:04:34.836585 16926 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:40387: Network error: Client connection negotiation failed: client connection to 127.3.115.60:40387: connect: Connection refused (error 111)
I20260620 06:04:34.838088 17007 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.60:40387
I20260620 06:04:34.838177 17075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.60:40387 every 8 connection(s)
I20260620 06:04:34.839182 17007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
I20260620 06:04:34.840094 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17007
I20260620 06:04:34.840225 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781935311537307-3532-0/minicluster-data/master-2/wal/instance
I20260620 06:04:34.840539 3532 external_mini_cluster.cc:1428] Setting key 31baf3304682e587faea209a83efb2ec
I20260620 06:04:34.842826 17076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:34.846895 17076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.848178 17001 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 3
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:34.857847 17079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.841992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36452 (local address 127.3.115.60:40387)
0620 06:04:34.842454 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.842470 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.842507 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:04:34.843335 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.843354 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.843720 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.844074 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.845048 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.846083 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.847016 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.847573 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.847797 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.850874 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.850881 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.850893 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.850924 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.854062 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.854694 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.854706 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.854714 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.854831 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.855133 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.855136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.855138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.855690 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.855918 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.856286 (+ 368us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.856633 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":142,"threads_started":1}
I20260620 06:04:34.860000 17005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.848650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36468)
0620 06:04:34.848943 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.848961 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.849061 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.850075 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.850078 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.850304 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.850417 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.850424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.851436 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.851441 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.852247 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.852260 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.852802 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.852805 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.852898 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.853293 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.853305 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.854846 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.857372 (+ 2526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.857375 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.857377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.857626 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.858982 (+ 1356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.858985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.858987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.859096 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.859505 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.859509 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.859620 (+ 111us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.859740 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.859878 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260620 06:04:34.860584 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.848728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36468 (local address 127.3.115.60:40387)
0620 06:04:34.849742 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.849752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.849766 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:04:34.849826 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.849838 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.849921 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.850025 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.850549 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.851307 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.852377 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.852598 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.852767 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.854986 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.854992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.854995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.855020 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.857247 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.858032 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.858037 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.858040 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.858159 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.859198 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.859202 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.859207 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.859407 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.859517 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.860123 (+ 606us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.860343 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":933,"thread_start_us":111,"threads_started":1}
I20260620 06:04:34.861341 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.849819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49590 (local address 127.3.115.62:36199)
0620 06:04:34.849941 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.849946 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.849964 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:04:34.850019 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.850022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.850070 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.850156 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.851119 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.851761 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.853506 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.853907 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.854031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.857231 (+ 3200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.857236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.857238 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.857268 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.858746 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.859275 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.859279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.859281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.859371 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.859654 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.859658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.859660 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.859918 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.860009 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.861057 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.861183 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:04:34.861357 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.848892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:36199 (local address 127.0.0.1:49590)
0620 06:04:34.849282 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.849336 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.849450 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.850338 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.850344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.850827 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.850976 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.850983 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.851876 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.851880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.853368 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.853381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.854033 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.854036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.854160 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.855042 (+ 882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.855057 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.857066 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.858865 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.858877 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.858887 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.859178 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.859464 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.859470 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.859473 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.859559 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.860609 (+ 1050us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.860618 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.860796 (+ 178us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.860990 (+ 194us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.861183 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":142,"threads_started":1}
I20260620 06:04:34.862512 17076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.864637 17001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24: Bootstrap starting.
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:34.867604 17001 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:34.868606 17001 log.cc:824] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:04:34.871392 17001 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24: No bootstrap required, opened a new log
I20260620 06:04:34.872042 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.861679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.3.115.62:36199)
0620 06:04:34.861794 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.861800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.861811 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:04:34.861844 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.861847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.861890 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.861974 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.862818 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.863349 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.864415 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.864785 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.864884 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.867942 (+ 3058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.867946 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.867949 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.867978 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.869765 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.870274 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.870279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.870281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.870379 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.870713 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.870716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.870718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.870936 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.871156 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.871719 (+ 563us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.871861 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:04:34.873107 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.863640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:36525 (local address 127.0.0.1:41712)
0620 06:04:34.863808 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.863823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.863906 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.864146 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.864149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.864324 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.864412 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.864418 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.865351 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.865355 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.866403 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.866415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.867123 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.867126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.867223 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.867767 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.867786 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.869292 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.871263 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.871266 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.871268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.871583 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.871920 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.871924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.871926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.872005 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.872492 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.872495 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.872602 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.872777 (+ 175us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.872942 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260620 06:04:34.873147 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.863700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41712 (local address 127.3.115.61:36525)
0620 06:04:34.863810 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.863815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.863866 (+ 51us) server_negotiation.cc:407] Connection header received
0620 06:04:34.864007 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.864010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.864057 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.864133 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.864521 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.865241 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.866542 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.866884 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.866954 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.869451 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.869455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.869457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.869484 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.871149 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.871706 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.871710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.871712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.871797 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.872109 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.872112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.872114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.872380 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.872488 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.872849 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.872985 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:04:34.874094 17076 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:04:34.876947 17001 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:34.877252 17001 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:34.877332 17001 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d8b819e63a74efc85dab06236b0da24, State: Initialized, Role: FOLLOWER
I20260620 06:04:34.877935 17001 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:04:34.879453 17001 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 06:04:34 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
I20260620 06:04:34.883591 17081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.874619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36472)
0620 06:04:34.875037 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:34.875051 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:34.875140 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:34.875380 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:34.875383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:34.875565 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:34.875662 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.875669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.876650 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.876684 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.877421 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:34.877431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.877933 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.877936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:34.878008 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.878473 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:34.878488 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:34.880214 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:34.881992 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.881998 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.882000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.882255 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.882520 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:34.882522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:34.882524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:34.882597 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:34.883067 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:34.883069 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:34.883146 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 06:04:34.883280 (+ 134us) client_negotiation.cc:240] Negotiation successful
0620 06:04:34.883427 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354}
I20260620 06:04:34.883852 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.873763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.3.115.61:36525)
0620 06:04:34.873866 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.873872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.873884 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:04:34.874093 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.874096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.874146 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.874217 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.874934 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.875629 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.876419 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.876713 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.876808 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.879881 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.879888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.879890 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.879919 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.881794 (+ 1875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.882377 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.882381 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.882383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.882473 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.882811 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.882817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.882821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.883030 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.883201 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.883582 (+ 381us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.883710 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:04:34.884963 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.874924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.3.115.60:40387)
0620 06:04:34.875036 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.875040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.875097 (+ 57us) server_negotiation.cc:407] Connection header received
0620 06:04:34.875226 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.875229 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.875273 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.875350 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.875778 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.876532 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.877552 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.877830 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.877934 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.880324 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.880327 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.880330 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.880356 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.881906 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.882363 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.882366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.882368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.882439 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.882675 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.882677 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.882679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.882944 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.883035 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.884455 (+ 1420us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.884832 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 06:04:34.885751 17094 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:04:34.885893 17094 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:04:34.883919 16933 master_runner.cc:186] Error getting master registration for 127.3.115.62:36199: OK, instance_id { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" instance_seqno: 1781935474357824 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:04:34.887689 16933 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:04:34.888283 17083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:34.888542 17083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:04:34.888550 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: Bootstrap starting.
I20260620 06:04:34.889240 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:04:34.891690 17076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:34.892807 17076 log.cc:824] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:34.901614 16929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.892059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49610 (local address 127.3.115.62:36199)
0620 06:04:34.892274 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.892278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.892289 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:04:34.892329 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.892331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.892370 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.892459 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.893447 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.894148 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.894976 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.895350 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.895474 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.897293 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.897297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.897299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.897331 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.899085 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.899563 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.899568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.899570 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.900041 (+ 471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.900403 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.900406 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.900408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.900601 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.900767 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.901164 (+ 397us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.901333 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260620 06:04:34.903424 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.893353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36478 (local address 127.3.115.60:40387)
0620 06:04:34.893452 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.893457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.893733 (+ 276us) server_negotiation.cc:407] Connection header received
0620 06:04:34.893912 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.893915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.893962 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.894018 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.894999 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.895775 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.897009 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.897365 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.897444 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.899184 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.899188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.899192 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.899235 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.900945 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.901442 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.901446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.901447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.901537 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.902392 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.902397 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.902401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.902621 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.902726 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.903098 (+ 372us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.903236 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 06:04:34.905851 17076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: No bootstrap required, opened a new log
I20260620 06:04:34.905938 17004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:34.892627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.3.115.61:36525)
0620 06:04:34.892785 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:34.892791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:34.892814 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:04:34.892858 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:34.892862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:34.892917 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:34.893002 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:34.895820 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.897017 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.898993 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:34.899345 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:34.899426 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:34.901716 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:34.901719 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:34.901722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:34.901758 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:34.903879 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.904387 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.904391 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.904393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.904483 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:34.905012 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:34.905016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:34.905018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:34.905205 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:34.905297 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:34.905620 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 06:04:34.905751 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 06:04:34.910651 17076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:34.911031 17076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:34.911098 17076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166ff9baf2b647ad95ce7dcd03521aff, State: Initialized, Role: FOLLOWER
I20260620 06:04:34.911665 17076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:34.912375 17104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:34.912542 17104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:04:34.913534 17076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: configured and running, proceeding with master startup.
W20260620 06:04:34.920110 17115 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:04:34.920226 17115 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:04:34.918627 17007 master_runner.cc:186] Error getting master registration for 127.3.115.62:36199: OK, instance_id { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" instance_seqno: 1781935474357824 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:04:34.922577 17007 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:04:34.938357 17083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:34.938616 17083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:34.939888 17083 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b6dedd9a534266b0770e48e2f4993b (127.3.115.62:36199), 166ff9baf2b647ad95ce7dcd03521aff (127.3.115.60:40387)
I20260620 06:04:34.940884 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8b819e63a74efc85dab06236b0da24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166ff9baf2b647ad95ce7dcd03521aff" is_pre_election: true
I20260620 06:04:34.941274 17050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d8b819e63a74efc85dab06236b0da24 in term 0.
I20260620 06:04:34.941964 16952 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166ff9baf2b647ad95ce7dcd03521aff, 8d8b819e63a74efc85dab06236b0da24; no voters:
I20260620 06:04:34.942405 17083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:04:34.942528 17083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:04:34.942577 17083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:34.944008 17083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:34.944447 17083 leader_election.cc:288] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [CANDIDATE]: Term 1 election: Requested vote from peers 88b6dedd9a534266b0770e48e2f4993b (127.3.115.62:36199), 166ff9baf2b647ad95ce7dcd03521aff (127.3.115.60:40387)
I20260620 06:04:34.944964 17050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8b819e63a74efc85dab06236b0da24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166ff9baf2b647ad95ce7dcd03521aff"
I20260620 06:04:34.945111 17050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:34.946542 17050 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d8b819e63a74efc85dab06236b0da24 in term 1.
I20260620 06:04:34.946929 16952 leader_election.cc:302] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166ff9baf2b647ad95ce7dcd03521aff, 8d8b819e63a74efc85dab06236b0da24; no voters:
I20260620 06:04:34.947129 17083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:34.947381 17083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [term 1 LEADER]: Becoming Leader. State: Replica: 8d8b819e63a74efc85dab06236b0da24, State: Running, Role: LEADER
I20260620 06:04:34.947793 17083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:34.950338 17117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8b819e63a74efc85dab06236b0da24. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:34.950497 17117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:34.952446 17119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:34.953519 17119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:34.962654 17050 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [term 1 FOLLOWER]: Refusing update from remote peer 8d8b819e63a74efc85dab06236b0da24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:34.963555 17083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:34.968778 17104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8b819e63a74efc85dab06236b0da24. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:34.968935 17104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:04:34.974020 17117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:34.974190 17117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:34.975559 17104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:34.975701 17104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:04:34.977118 17120 mvcc.cc:205] Tried to move back new op lower bound from 7298807705432264704 to 7298807705389903872. Current Snapshot: MvccSnapshot[applied={T|T < 7298807705432264704}]
I20260620 06:04:34.977625 17121 mvcc.cc:205] Tried to move back new op lower bound from 7298807705432264704 to 7298807705389903872. Current Snapshot: MvccSnapshot[applied={T|T < 7298807705432264704}]
I20260620 06:04:34.983110 17119 catalog_manager.cc:1347] Generated new cluster ID: 8650b5500dbe4946899a98b14eeb06e0
I20260620 06:04:34.983222 17119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:35.000828 16926 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } attempt: 4
Jun 20 06:04:35 dist-test-slave-4h14 krb5kdc[16845](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935474, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
I20260620 06:04:35.014418 16930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:35.001465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:40387 (local address 127.0.0.1:36482)
0620 06:04:35.001752 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:35.001768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:35.001864 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:35.002483 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:35.002486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:35.002723 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:35.002842 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:35.002850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:35.004120 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:35.004124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:35.005142 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:35.005156 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:35.005961 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:35.005966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:35.006056 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:35.006789 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:35.006805 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:35.009607 (+ 2802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:35.012269 (+ 2662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:35.012273 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:35.012275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:35.012579 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:35.013025 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:35.013028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:35.013046 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:35.013150 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:35.013793 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:35.013798 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:35.013915 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 06:04:35.014023 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 06:04:35.014192 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260620 06:04:35.015416 17080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:35.002029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36482 (local address 127.3.115.60:40387)
0620 06:04:35.002150 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:35.002155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:35.002168 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:04:35.002220 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:35.002223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:35.002271 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:35.002341 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:35.002983 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:35.003810 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:35.005291 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:35.005654 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:35.005784 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:35.009824 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:35.009828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:35.009831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:35.009860 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:35.012066 (+ 2206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:35.012778 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:35.012783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:35.012785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:35.012888 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:35.013296 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:35.013300 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:35.013319 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:35.013577 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:35.013689 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:35.015094 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0620 06:04:35.015244 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:04:35.017644 17119 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:04:35.019310 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: Bootstrap starting.
I20260620 06:04:35.020993 17119 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:35.022405 16926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:35.023403 16926 log.cc:824] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:35.029608 16926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: No bootstrap required, opened a new log
I20260620 06:04:35.037276 16926 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:35.041049 16926 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:35.041281 16926 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b6dedd9a534266b0770e48e2f4993b, State: Initialized, Role: FOLLOWER
I20260620 06:04:35.042220 16926 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } }
I20260620 06:04:35.045060 16926 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:35.045297 16900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8b819e63a74efc85dab06236b0da24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b6dedd9a534266b0770e48e2f4993b" is_pre_election: true
I20260620 06:04:35.045691 16900 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d8b819e63a74efc85dab06236b0da24 in term 0.
I20260620 06:04:35.046216 16898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:35.047904 16898 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [term 1 FOLLOWER]: Refusing update from remote peer 8d8b819e63a74efc85dab06236b0da24: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:35.048910 17083 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:35.050127 17119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24: Generated new TSK 0
I20260620 06:04:35.048727 16899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8d8b819e63a74efc85dab06236b0da24" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b6dedd9a534266b0770e48e2f4993b"
I20260620 06:04:35.051117 17119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:04:35.064026 17138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:35.064193 17138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:04:35.064301 17138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:35.064378 17138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:04:35.064824 17126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d8b819e63a74efc85dab06236b0da24. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:35.064944 17126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 06:04:35.075229 17137 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:04:35.075368 17137 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:04:35.078974 17139 mvcc.cc:205] Tried to move back new op lower bound from 7298807705770061824 to 7298807705389903872. Current Snapshot: MvccSnapshot[applied={T|T < 7298807705432264704}]
I20260620 06:04:35.479012 17083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d8b819e63a74efc85dab06236b0da24" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b6dedd9a534266b0770e48e2f4993b" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 36199 } } peers { permanent_uuid: "8d8b819e63a74efc85dab06236b0da24" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 36525 } } peers { permanent_uuid: "166ff9baf2b647ad95ce7dcd03521aff" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 40387 } } }
I20260620 06:04:35.479246 17083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d8b819e63a74efc85dab06236b0da24 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:35.921659 17115 catalog_manager.cc:1259] Loaded cluster ID: 8650b5500dbe4946899a98b14eeb06e0
I20260620 06:04:35.921762 17115 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: loading cluster ID for follower catalog manager: success
I20260620 06:04:35.923504 17115 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: acquiring CA information for follower catalog manager: success
I20260620 06:04:35.924831 17115 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 166ff9baf2b647ad95ce7dcd03521aff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:04:36.076833 17137 catalog_manager.cc:1259] Loaded cluster ID: 8650b5500dbe4946899a98b14eeb06e0
I20260620 06:04:36.076956 17137 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: loading cluster ID for follower catalog manager: success
I20260620 06:04:36.078701 17137 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: acquiring CA information for follower catalog manager: success
I20260620 06:04:36.080394 17137 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 88b6dedd9a534266b0770e48e2f4993b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:04:36.139523 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16858
I20260620 06:04:36.145920 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 16933
I20260620 06:04:36.152288 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17007
2026-06-20T06:04:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3198 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:04:36 dist-test-slave-4h14 krb5kdc[17152](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:04:36 dist-test-slave-4h14 krb5kdc[17152](info): set up 2 sockets
Jun 20 06:04:36 dist-test-slave-4h14 krb5kdc[17152](info): commencing operation
krb5kdc: starting...
W20260620 06:04:37.197132 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:04:37 dist-test-slave-4h14 krb5kdc[17152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:04:37.332526 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40831
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:40831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:37.453164 17165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:37.453495 17165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:37.453540 17165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:37.457746 17165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:37.457819 17165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:37.457840 17165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:37.457857 17165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:37.457873 17165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:37.463399 17165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:40831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:40831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:37.464481 17165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:37.465715 17165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:37.474617 17173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:37.474628 17170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:37.474716 17171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:37.475158 17165 server_base.cc:1034] running on GCE node
I20260620 06:04:37.475761 17165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:37.477133 17165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:37.478302 17165 hybrid_clock.cc:648] HybridClock initialized: now 1781935477478264 us; error 64 us; skew 500 ppm
Jun 20 06:04:37 dist-test-slave-4h14 krb5kdc[17152](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935477, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:37.481292 17165 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:37.482255 17165 webserver.cc:460] Webserver started at http://127.3.115.62:46133/ using document root <none> and password file <none>
I20260620 06:04:37.482800 17165 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:37.482862 17165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:37.483032 17165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:37.485258 17165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "4c7587eb36814204a70d3fb710f94933"
format_stamp: "Formatted at 2026-06-20 06:04:37 on dist-test-slave-4h14"
server_key: "273b90bcf08e4da5ec1a850305ab1e98"
server_key_iv: "0021fd10f181975205330cb41b666533"
server_key_version: "encryptionkey@0"
I20260620 06:04:37.485811 17165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "4c7587eb36814204a70d3fb710f94933"
format_stamp: "Formatted at 2026-06-20 06:04:37 on dist-test-slave-4h14"
server_key: "273b90bcf08e4da5ec1a850305ab1e98"
server_key_iv: "0021fd10f181975205330cb41b666533"
server_key_version: "encryptionkey@0"
I20260620 06:04:37.490324 17165 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:04:37.493086 17180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:37.494488 17165 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:37.494602 17165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "4c7587eb36814204a70d3fb710f94933"
format_stamp: "Formatted at 2026-06-20 06:04:37 on dist-test-slave-4h14"
server_key: "273b90bcf08e4da5ec1a850305ab1e98"
server_key_iv: "0021fd10f181975205330cb41b666533"
server_key_version: "encryptionkey@0"
I20260620 06:04:37.494777 17165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:37.520498 17165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:37.524247 17165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:37.524423 17165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:37.532522 17165 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:40831
I20260620 06:04:37.532612 17232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:40831 every 8 connection(s)
I20260620 06:04:37.533663 17165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:37.536902 17233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:37.540026 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17165
I20260620 06:04:37.540167 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:04:37.540512 3532 external_mini_cluster.cc:1428] Setting key 0d11ba96daa4678fc630af292f8134b2
I20260620 06:04:37.543730 17233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933: Bootstrap starting.
I20260620 06:04:37.546576 17233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:37.547391 17233 log.cc:824] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:37.549507 17233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933: No bootstrap required, opened a new log
Jun 20 06:04:37 dist-test-slave-4h14 krb5kdc[17152](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935477, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:37.553176 17233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7587eb36814204a70d3fb710f94933" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40831 } }
I20260620 06:04:37.553490 17233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:37.553574 17233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c7587eb36814204a70d3fb710f94933, State: Initialized, Role: FOLLOWER
I20260620 06:04:37.554055 17233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7587eb36814204a70d3fb710f94933" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40831 } }
I20260620 06:04:37.554224 17233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:37.554312 17233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:37.554437 17233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:37.555429 17233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7587eb36814204a70d3fb710f94933" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40831 } }
I20260620 06:04:37.555809 17233 leader_election.cc:302] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4c7587eb36814204a70d3fb710f94933; no voters:
I20260620 06:04:37.556097 17233 leader_election.cc:288] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:04:37.556221 17238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:37.556463 17238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [term 1 LEADER]: Becoming Leader. State: Replica: 4c7587eb36814204a70d3fb710f94933, State: Running, Role: LEADER
I20260620 06:04:37.556828 17236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:37.542036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33302 (local address 127.3.115.62:40831)
0620 06:04:37.542513 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:37.542531 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:37.542571 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:04:37.543236 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:37.543255 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:37.543614 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:37.543987 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:37.545037 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:37.546180 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:37.547184 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:37.547740 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:37.547914 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:37.550611 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:37.550617 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:37.550633 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:37.550669 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:37.553090 (+ 2421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:37.553635 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:37.553643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:37.553653 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:37.553876 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:37.554167 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:37.554170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:37.554172 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:37.554761 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:37.555005 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:37.555255 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 06:04:37.555612 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":164,"threads_started":1}
I20260620 06:04:37.557726 17233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:37.556820 17238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7587eb36814204a70d3fb710f94933" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40831 } }
I20260620 06:04:37.559902 17239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4c7587eb36814204a70d3fb710f94933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7587eb36814204a70d3fb710f94933" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40831 } } }
I20260620 06:04:37.560479 17239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:37.560194 17240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4c7587eb36814204a70d3fb710f94933. Latest consensus state: current_term: 1 leader_uuid: "4c7587eb36814204a70d3fb710f94933" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4c7587eb36814204a70d3fb710f94933" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 40831 } } }
I20260620 06:04:37.560628 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:04:37.560705 17240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933 [sys.catalog]: This master's current role is: LEADER
W20260620 06:04:37.563022 17252 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:04:37.563120 17252 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:04:37.563231 17253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:37.564090 17253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:37.569949 17253 catalog_manager.cc:1347] Generated new cluster ID: c1ee948447c94f07b678c50e81e8aab2
I20260620 06:04:37.570044 17253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:37.601272 17253 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:04:37.602638 17253 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:37.614980 17253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4c7587eb36814204a70d3fb710f94933: Generated new TSK 0
I20260620 06:04:37.615957 17253 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:04:37.668964 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17165
2026-06-20T06:04:37Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1515 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:04:37 dist-test-slave-4h14 krb5kdc[17260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:04:37 dist-test-slave-4h14 krb5kdc[17260](info): set up 2 sockets
Jun 20 06:04:37 dist-test-slave-4h14 krb5kdc[17260](info): commencing operation
krb5kdc: starting...
W20260620 06:04:38.713517 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:04:38 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:04:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:04:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:04:38.844558 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37891
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:38.959031 17273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:38.959420 17273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:38.959515 17273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:38.963821 17273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:38.963945 17273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:38.963984 17273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:38.964025 17273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:38.964061 17273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:38.969781 17273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:38.970994 17273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:38.972100 17273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:38.981359 17279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:38.981353 17278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:38.981410 17273 server_base.cc:1034] running on GCE node
W20260620 06:04:38.981353 17281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:38.982254 17273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:38.983428 17273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:38.984722 17273 hybrid_clock.cc:648] HybridClock initialized: now 1781935478984632 us; error 39 us; skew 500 ppm
Jun 20 06:04:38 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935478, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:38.991524 17273 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:38.992560 17273 webserver.cc:460] Webserver started at http://127.3.115.62:44785/ using document root <none> and password file <none>
I20260620 06:04:38.993204 17273 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:38.993314 17273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:38.993577 17273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:38.995795 17273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "8eb29e75041f4e678c044a155b9f5712"
format_stamp: "Formatted at 2026-06-20 06:04:38 on dist-test-slave-4h14"
server_key: "1cbbcbf082651d6e990d8708d99d40b3"
server_key_iv: "8e23b9bb94298c0a53ad23d631517f77"
server_key_version: "encryptionkey@0"
I20260620 06:04:38.996436 17273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "8eb29e75041f4e678c044a155b9f5712"
format_stamp: "Formatted at 2026-06-20 06:04:38 on dist-test-slave-4h14"
server_key: "1cbbcbf082651d6e990d8708d99d40b3"
server_key_iv: "8e23b9bb94298c0a53ad23d631517f77"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.000561 17273 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260620 06:04:39.003357 17288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.004928 17273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:39.005089 17273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "8eb29e75041f4e678c044a155b9f5712"
format_stamp: "Formatted at 2026-06-20 06:04:38 on dist-test-slave-4h14"
server_key: "1cbbcbf082651d6e990d8708d99d40b3"
server_key_iv: "8e23b9bb94298c0a53ad23d631517f77"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.005242 17273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:39.026243 17273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:39.030042 17273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:39.030283 17273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:39.037858 17273 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:37891
I20260620 06:04:39.038158 17340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:37891 every 8 connection(s)
I20260620 06:04:39.039040 17273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:39.041656 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17273
I20260620 06:04:39.041851 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:04:39.042176 3532 external_mini_cluster.cc:1428] Setting key 3691e1daa84f3744b327ad22f3b76a99
I20260620 06:04:39.042181 17341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:39.049363 17341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap starting.
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:39.052031 17341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:39.053002 17341 log.cc:824] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:39.055661 17341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: No bootstrap required, opened a new log
I20260620 06:04:39.058413 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.043904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50032 (local address 127.3.115.62:37891)
0620 06:04:39.044401 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.044417 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.044458 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:04:39.045240 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.045262 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.045636 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.046071 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:39.046931 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.047956 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.048930 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.049431 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.049617 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.052506 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:39.052512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:39.052531 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:39.052567 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:39.054929 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.055427 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.055433 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.055439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.055584 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.055886 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.055889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.055891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.056447 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:39.056644 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.056961 (+ 317us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.057291 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":153,"threads_started":1}
I20260620 06:04:39.060413 17341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:39.060726 17341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:39.060807 17341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb29e75041f4e678c044a155b9f5712, State: Initialized, Role: FOLLOWER
I20260620 06:04:39.061231 17341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:39.061359 17341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:39.061451 17341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:39.061569 17341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:39.062558 17341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:39.062937 17341 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb29e75041f4e678c044a155b9f5712; no voters:
I20260620 06:04:39.063292 17341 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:04:39.063462 17346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:39.063714 17346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb29e75041f4e678c044a155b9f5712, State: Running, Role: LEADER
I20260620 06:04:39.064045 17346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:39.064638 17341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:39.065881 17348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb29e75041f4e678c044a155b9f5712. Latest consensus state: current_term: 1 leader_uuid: "8eb29e75041f4e678c044a155b9f5712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } } }
I20260620 06:04:39.066015 17348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:39.066192 17347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8eb29e75041f4e678c044a155b9f5712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } } }
I20260620 06:04:39.066349 17347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:39.067214 17355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:39.069952 17355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:39.075992 17355 catalog_manager.cc:1347] Generated new cluster ID: 79b26b70ecbf4ecb96df586d1c243f21
I20260620 06:04:39.076077 17355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:39.086190 17355 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:04:39.087539 17355 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:39.097395 17355 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Generated new TSK 0
I20260620 06:04:39.098418 17355 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:04:39.131443 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:37891
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:39.251437 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:39.251744 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:39.251796 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:39.256029 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:39.256114 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:39.256222 17367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:04:39.261830 17367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:39.263021 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:39.264134 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:39.274469 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:39.274478 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:39.274478 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:39.274809 17367 server_base.cc:1034] running on GCE node
I20260620 06:04:39.275247 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:39.275982 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:39.277180 17367 hybrid_clock.cc:648] HybridClock initialized: now 1781935479277164 us; error 31 us; skew 500 ppm
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935479, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:39.280309 17367 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:04:39.281641 17367 webserver.cc:460] Webserver started at http://127.3.115.1:46387/ using document root <none> and password file <none>
I20260620 06:04:39.282353 17367 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:39.282428 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:39.282642 17367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:39.285380 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "7c9a49ba65ee8607e92bd6c27ab2027d"
server_key_iv: "35feea0cd61403db23ed32826e63090e"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.286080 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "7c9a49ba65ee8607e92bd6c27ab2027d"
server_key_iv: "35feea0cd61403db23ed32826e63090e"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.291114 17367 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:04:39.293799 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.295219 17367 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:39.295387 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "7c9a49ba65ee8607e92bd6c27ab2027d"
server_key_iv: "35feea0cd61403db23ed32826e63090e"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.295513 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:39.313053 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:39.316232 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:39.316484 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:39.317214 17367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:39.318066 17367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:39.318141 17367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.318214 17367 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:39.318259 17367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.327062 17367 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:34795
I20260620 06:04:39.327117 17495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:34795 every 8 connection(s)
I20260620 06:04:39.328032 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:04:39.329591 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17367
I20260620 06:04:39.329728 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:04:39.330111 3532 external_mini_cluster.cc:1428] Setting key 56b063904fc4ac2dc301fce850982857
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935479, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:04:39.343698 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.330107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:42013 (local address 127.3.115.62:37891)
0620 06:04:39.330278 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.330284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.331365 (+ 1081us) server_negotiation.cc:407] Connection header received
0620 06:04:39.332982 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.332989 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.333053 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.333151 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:39.334454 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.335303 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.336634 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.337116 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.337255 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.339875 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:39.339879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:39.339883 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:39.339916 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:39.341369 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.341862 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.341867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.341869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.341960 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.342279 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.342282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.342284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.342514 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:39.342616 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.343281 (+ 665us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.343439 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:04:39.344470 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.330583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:42013)
0620 06:04:39.331173 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:39.331235 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:39.332743 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:39.333342 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:39.333350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:39.333777 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:39.334292 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.334306 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.335453 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.335456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.336491 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.336501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.337311 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.337314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.337469 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.338160 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:39.338177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:39.339627 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:39.341476 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:39.341481 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:39.341495 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:39.341764 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:39.342050 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:39.342052 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:39.342053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:39.342185 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:39.342650 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:39.342660 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:39.342986 (+ 326us) client_negotiation.cc:769] Sending connection context
0620 06:04:39.343155 (+ 169us) client_negotiation.cc:240] Negotiation successful
0620 06:04:39.343446 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":139,"threads_started":1}
I20260620 06:04:39.345899 17496 heartbeater.cc:347] Connected to a master server at 127.3.115.62:37891
I20260620 06:04:39.346164 17496 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:39.346733 17496 heartbeater.cc:511] Master 127.3.115.62:37891 requested a full tablet report, sending...
I20260620 06:04:39.348278 17305 ts_manager.cc:194] Registered new tserver with Master: 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795)
I20260620 06:04:39.349481 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:42013
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:04:39.359368 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:37891
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:39.473552 17501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:39.473861 17501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:39.473945 17501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:39.477983 17501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:39.478080 17501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:39.478175 17501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:04:39.483503 17501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:39.484640 17501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:39.485790 17501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:39.495749 17509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:39.495734 17506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:39.495916 17507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:39.496019 17501 server_base.cc:1034] running on GCE node
I20260620 06:04:39.496448 17501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:39.497172 17501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:39.498441 17501 hybrid_clock.cc:648] HybridClock initialized: now 1781935479498391 us; error 36 us; skew 500 ppm
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935479, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:39.501421 17501 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:04:39.502292 17501 webserver.cc:460] Webserver started at http://127.3.115.2:37701/ using document root <none> and password file <none>
I20260620 06:04:39.502818 17501 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:39.502877 17501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:39.503041 17501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:39.505204 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "edd17449f93d41349906e355b3c3a5be"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "2639a29820c70481723a98cb26af3acf"
server_key_iv: "4c30c1796ac6a858fce05fdfe28a25e1"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.505764 17501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "edd17449f93d41349906e355b3c3a5be"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "2639a29820c70481723a98cb26af3acf"
server_key_iv: "4c30c1796ac6a858fce05fdfe28a25e1"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.509935 17501 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:04:39.512553 17516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.514094 17501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:04:39.514215 17501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "edd17449f93d41349906e355b3c3a5be"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "2639a29820c70481723a98cb26af3acf"
server_key_iv: "4c30c1796ac6a858fce05fdfe28a25e1"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.514364 17501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:39.572014 17501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:39.575800 17501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:39.576026 17501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:39.576726 17501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:39.577618 17501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:39.577693 17501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.577762 17501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:39.577804 17501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.586977 17501 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:33621
I20260620 06:04:39.587042 17629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:33621 every 8 connection(s)
I20260620 06:04:39.587975 17501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:04:39.597193 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17501
I20260620 06:04:39.597399 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:04:39.597709 3532 external_mini_cluster.cc:1428] Setting key 0c1388b20aed2eab5810b2e10c8510e5
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935479, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:39.603192 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.590034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:45233 (local address 127.3.115.62:37891)
0620 06:04:39.590172 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.590176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.591087 (+ 911us) server_negotiation.cc:407] Connection header received
0620 06:04:39.592358 (+ 1271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.592361 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.592412 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.592493 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:39.593876 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.594423 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.595645 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.595886 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.596007 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.599191 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:39.599200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:39.599202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:39.599226 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:39.600980 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.601558 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.601563 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.601565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.601650 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.602009 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.602014 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.602016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.602211 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:39.602319 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.602893 (+ 574us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.603027 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:04:39.604102 17632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.590405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:45233)
0620 06:04:39.590906 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:39.590958 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:39.592132 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:39.592762 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:39.592770 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:39.593164 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:39.593691 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.593707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.594571 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.594580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.595509 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.595519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.596053 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.596057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.596238 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.597043 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:39.597067 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:39.599007 (+ 1940us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:39.601106 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:39.601113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:39.601132 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:39.601445 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:39.601742 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:39.601745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:39.601747 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:39.601911 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:39.602329 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:39.602338 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:39.602614 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 06:04:39.602825 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:04:39.603187 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":103,"threads_started":1}
I20260620 06:04:39.605639 17630 heartbeater.cc:347] Connected to a master server at 127.3.115.62:37891
I20260620 06:04:39.606009 17630 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:39.606629 17630 heartbeater.cc:511] Master 127.3.115.62:37891 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:04:39.607798 17305 ts_manager.cc:194] Registered new tserver with Master: edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621)
I20260620 06:04:39.608412 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:45233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:04:39.625748 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:37891
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:39.740046 17635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:39.740340 17635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:39.740386 17635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:39.744524 17635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:39.744593 17635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:39.744709 17635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:04:39.750130 17635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:39.751242 17635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:39.752352 17635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:39.762368 17641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:39.762353 17643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:39.762395 17640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:39.762831 17635 server_base.cc:1034] running on GCE node
I20260620 06:04:39.763464 17635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:39.764159 17635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:39.765372 17635 hybrid_clock.cc:648] HybridClock initialized: now 1781935479765305 us; error 97 us; skew 500 ppm
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935479, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:39.768441 17635 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:04:39.769414 17635 webserver.cc:460] Webserver started at http://127.3.115.3:37427/ using document root <none> and password file <none>
I20260620 06:04:39.770000 17635 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:39.770063 17635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:39.770288 17635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:39.772450 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "5c38c30657914004b51cc15231259f02"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "14e222b40e38f26d8c8f7a7dcc171ce2"
server_key_iv: "e8628ea3c7803e4b9180fb765c879b5f"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.773140 17635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "5c38c30657914004b51cc15231259f02"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "14e222b40e38f26d8c8f7a7dcc171ce2"
server_key_iv: "e8628ea3c7803e4b9180fb765c879b5f"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.777001 17635 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:04:39.779582 17650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.780948 17635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:39.781085 17635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "5c38c30657914004b51cc15231259f02"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "14e222b40e38f26d8c8f7a7dcc171ce2"
server_key_iv: "e8628ea3c7803e4b9180fb765c879b5f"
server_key_version: "encryptionkey@0"
I20260620 06:04:39.781205 17635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:39.798563 17635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:39.802069 17635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:39.802290 17635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:39.802929 17635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:39.803864 17635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:39.803936 17635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.804006 17635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:39.804054 17635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:39.813974 17635 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:44801
I20260620 06:04:39.814002 17763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:44801 every 8 connection(s)
I20260620 06:04:39.814966 17635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:04:39.822887 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17635
I20260620 06:04:39.823041 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:04:39.823321 3532 external_mini_cluster.cc:1428] Setting key 3ec8089e2412d847a6a55057e63d36c8
Jun 20 06:04:39 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935479, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:39.829491 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.816998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:36061 (local address 127.3.115.62:37891)
0620 06:04:39.817141 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.817144 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.818075 (+ 931us) server_negotiation.cc:407] Connection header received
0620 06:04:39.819451 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.819454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.819517 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.819623 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:39.820867 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.821412 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.822565 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.822816 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.822931 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.826073 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:39.826076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:39.826078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:39.826104 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:39.827410 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.827944 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.827947 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.827948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.828023 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.828392 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.828394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.828395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.828567 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:39.828638 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.829250 (+ 612us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.829367 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:04:39.830380 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.817410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:36061)
0620 06:04:39.817864 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:39.817939 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:39.819237 (+ 1298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:39.819796 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:39.819804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:39.820181 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:39.820717 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.820739 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.821540 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.821543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.822440 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.822449 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.822973 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.822977 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.823198 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.823892 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:39.823918 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:39.825923 (+ 2005us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:39.827516 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:39.827521 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:39.827534 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:39.827849 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:39.828128 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:39.828130 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:39.828132 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:39.828299 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:39.828710 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:39.828724 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:39.828989 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 06:04:39.829166 (+ 177us) client_negotiation.cc:240] Negotiation successful
0620 06:04:39.829468 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260620 06:04:39.831612 17764 heartbeater.cc:347] Connected to a master server at 127.3.115.62:37891
I20260620 06:04:39.831888 17764 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:39.832413 17764 heartbeater.cc:511] Master 127.3.115.62:37891 requested a full tablet report, sending...
I20260620 06:04:39.833602 17305 ts_manager.cc:194] Registered new tserver with Master: 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801)
I20260620 06:04:39.834229 17305 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:36061
I20260620 06:04:39.837677 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:04:39.848228 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.839785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50034 (local address 127.3.115.62:37891)
0620 06:04:39.839977 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.839981 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.840126 (+ 145us) server_negotiation.cc:407] Connection header received
0620 06:04:39.840316 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.840324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.840367 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.840451 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:39.841252 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.841842 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.842719 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.842939 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.843046 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.844463 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:39.844466 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:39.844469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:39.844503 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:39.846253 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.846763 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.846769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.846771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.846868 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:39.847201 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:39.847205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:39.847207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:39.847465 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:39.847635 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.847890 (+ 255us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.848037 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260620 06:04:39.851817 17305 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:04:39.855119 17305 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:04:39.871196 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.865585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.3.115.3:44801)
0620 06:04:39.866139 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.866154 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.866659 (+ 505us) server_negotiation.cc:407] Connection header received
0620 06:04:39.866791 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.866795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.866998 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.867196 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:39.867380 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.868256 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.869893 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.870464 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.870590 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.870741 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.870930 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.871032 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":196,"threads_started":1}
I20260620 06:04:39.871196 17779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.865218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:44801 (local address 127.0.0.1:46782)
0620 06:04:39.866548 (+ 1330us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:39.866563 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:39.866676 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:39.867168 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:39.867171 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:39.867188 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:39.867278 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.867285 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.868391 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.868394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.869780 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.869787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.870603 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.870606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.870719 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.870760 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 06:04:39.870899 (+ 139us) client_negotiation.cc:240] Negotiation successful
0620 06:04:39.871043 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1257,"mutex_wait_us":187,"thread_start_us":94,"threads_started":1}
I20260620 06:04:39.871387 17775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.864351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:34795 (local address 127.0.0.1:49070)
0620 06:04:39.865250 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:39.865310 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:39.865471 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:39.866244 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:39.866252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:39.866279 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:39.866439 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.866448 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.867918 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.867922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.869453 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.869463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.870806 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.870810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.871017 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.871074 (+ 57us) client_negotiation.cc:769] Sending connection context
0620 06:04:39.871147 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:04:39.871241 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"mutex_wait_us":113,"spinlock_wait_cycles":63872,"thread_start_us":110,"threads_started":1}
I20260620 06:04:39.871814 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.864699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.3.115.2:33621)
0620 06:04:39.865113 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.865129 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.865625 (+ 496us) server_negotiation.cc:407] Connection header received
0620 06:04:39.865716 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.865720 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.865945 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.866093 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:39.866595 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.867786 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.869727 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.870497 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.870590 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.870740 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.871457 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.871673 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":86,"threads_started":1}
I20260620 06:04:39.872093 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.864506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49070 (local address 127.3.115.1:34795)
0620 06:04:39.865255 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:39.865263 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:39.865440 (+ 177us) server_negotiation.cc:407] Connection header received
0620 06:04:39.865695 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:39.865715 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:39.865945 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:39.866093 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:39.866553 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.867786 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.869578 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.870465 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.870597 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.870740 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:39.871576 (+ 836us) server_negotiation.cc:299] Negotiation successful
0620 06:04:39.871672 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":78,"threads_started":1}
I20260620 06:04:39.872143 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:39.864525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33621 (local address 127.0.0.1:45102)
0620 06:04:39.865270 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:39.865317 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:39.865471 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:39.866307 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:39.866312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:39.866331 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:39.866463 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.866469 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.867949 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.867952 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.869603 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:39.869613 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:39.870806 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:39.870810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:39.871021 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:39.871077 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 06:04:39.871147 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:04:39.872003 (+ 856us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":292,"thread_start_us":198,"threads_started":1}
I20260620 06:04:39.874558 17430 tablet_service.cc:1467] Processing CreateTablet for tablet 850d2f6420d24959b2a44d31117ec35f (DEFAULT_TABLE table=test-table [id=4164f15e91e8431897b7ec65b7b3e4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:39.874557 17697 tablet_service.cc:1467] Processing CreateTablet for tablet 850d2f6420d24959b2a44d31117ec35f (DEFAULT_TABLE table=test-table [id=4164f15e91e8431897b7ec65b7b3e4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:39.875542 17430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d2f6420d24959b2a44d31117ec35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:39.875553 17697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d2f6420d24959b2a44d31117ec35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:39.881807 17781 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Bootstrap starting.
I20260620 06:04:39.883257 17782 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap starting.
I20260620 06:04:39.884621 17781 tablet_bootstrap.cc:654] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:39.885690 17782 tablet_bootstrap.cc:654] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:39.885742 17781 log.cc:824] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:39.885543 17564 tablet_service.cc:1467] Processing CreateTablet for tablet 850d2f6420d24959b2a44d31117ec35f (DEFAULT_TABLE table=test-table [id=4164f15e91e8431897b7ec65b7b3e4cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:39.886588 17782 log.cc:824] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:39.886793 17564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850d2f6420d24959b2a44d31117ec35f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:39.888025 17781 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: No bootstrap required, opened a new log
I20260620 06:04:39.888249 17781 ts_tablet_manager.cc:1397] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:04:39.889302 17782 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: No bootstrap required, opened a new log
I20260620 06:04:39.889510 17782 ts_tablet_manager.cc:1397] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:04:39.891647 17781 raft_consensus.cc:348] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:39.891918 17781 raft_consensus.cc:374] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:39.891973 17781 raft_consensus.cc:729] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c38c30657914004b51cc15231259f02, State: Initialized, Role: FOLLOWER
I20260620 06:04:39.892488 17781 consensus_queue.cc:260] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:39.893419 17781 ts_tablet_manager.cc:1428] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:04:39.893810 17764 heartbeater.cc:503] Master 127.3.115.62:37891 was elected leader, sending a full tablet report...
I20260620 06:04:39.894137 17782 raft_consensus.cc:348] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:39.894428 17782 raft_consensus.cc:374] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:39.894483 17782 raft_consensus.cc:729] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6120ff42da8c49c2bbf9aa7b6966183f, State: Initialized, Role: FOLLOWER
I20260620 06:04:39.894768 17786 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Bootstrap starting.
I20260620 06:04:39.894958 17782 consensus_queue.cc:260] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:39.895936 17782 ts_tablet_manager.cc:1428] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:04:39.896322 17496 heartbeater.cc:503] Master 127.3.115.62:37891 was elected leader, sending a full tablet report...
I20260620 06:04:39.898094 17786 tablet_bootstrap.cc:654] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:39.898937 17786 log.cc:824] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:39.901017 17786 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: No bootstrap required, opened a new log
I20260620 06:04:39.901196 17786 ts_tablet_manager.cc:1397] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:04:39.904232 17786 raft_consensus.cc:348] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:39.904516 17786 raft_consensus.cc:374] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:39.904577 17786 raft_consensus.cc:729] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd17449f93d41349906e355b3c3a5be, State: Initialized, Role: FOLLOWER
I20260620 06:04:39.905047 17786 consensus_queue.cc:260] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:39.905838 17630 heartbeater.cc:503] Master 127.3.115.62:37891 was elected leader, sending a full tablet report...
I20260620 06:04:39.906034 17786 ts_tablet_manager.cc:1428] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:04:40.047749 17787 raft_consensus.cc:482] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:40.048009 17787 raft_consensus.cc:504] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:40.049418 17787 leader_election.cc:288] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621), 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801)
I20260620 06:04:40.053782 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.049808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:55867 (local address 127.3.115.2:33621)
0620 06:04:40.049950 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:40.049955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:40.050113 (+ 158us) server_negotiation.cc:407] Connection header received
0620 06:04:40.050321 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:40.050325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:40.050390 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:40.050452 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:40.050890 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.051399 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.052509 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.053109 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.053236 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.053308 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:40.053523 (+ 215us) server_negotiation.cc:299] Negotiation successful
0620 06:04:40.053614 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:04:40.053790 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.049668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33621 (local address 127.3.115.1:55867)
0620 06:04:40.050007 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:40.050024 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:40.050147 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:40.050615 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:40.050618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:40.050635 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:40.050771 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.050776 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.051526 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.051530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.052372 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.052381 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.053261 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.053264 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.053346 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.053369 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:04:40.053502 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 06:04:40.053587 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":149,"threads_started":1}
I20260620 06:04:40.053934 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.050352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:50073 (local address 127.3.115.3:44801)
0620 06:04:40.050487 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:40.050492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:40.050551 (+ 59us) server_negotiation.cc:407] Connection header received
0620 06:04:40.050719 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:40.050722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:40.050775 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:40.050896 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:40.051232 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.051712 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.052773 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.053376 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.053498 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.053559 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:40.053737 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 06:04:40.053815 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:04:40.054203 17791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.050146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:44801 (local address 127.3.115.1:50073)
0620 06:04:40.050485 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:40.050498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:40.050602 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:40.050919 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:40.050922 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:40.050942 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:40.051078 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.051084 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.051840 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.051844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.052607 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.052615 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.053510 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.053513 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.053610 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.053629 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:04:40.053704 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 06:04:40.053755 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":94,"threads_started":1}
I20260620 06:04:40.054587 17584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d2f6420d24959b2a44d31117ec35f" candidate_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd17449f93d41349906e355b3c3a5be" is_pre_election: true
I20260620 06:04:40.054986 17584 raft_consensus.cc:2454] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6120ff42da8c49c2bbf9aa7b6966183f in term 0.
I20260620 06:04:40.055366 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d2f6420d24959b2a44d31117ec35f" candidate_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c38c30657914004b51cc15231259f02" is_pre_election: true
I20260620 06:04:40.055549 17383 leader_election.cc:302] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6120ff42da8c49c2bbf9aa7b6966183f, edd17449f93d41349906e355b3c3a5be; no voters:
I20260620 06:04:40.055651 17718 raft_consensus.cc:2454] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6120ff42da8c49c2bbf9aa7b6966183f in term 0.
I20260620 06:04:40.055864 17787 raft_consensus.cc:2781] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:04:40.055998 17787 raft_consensus.cc:482] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:04:40.056062 17787 raft_consensus.cc:3037] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:40.057209 17787 raft_consensus.cc:504] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:40.057586 17787 leader_election.cc:288] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [CANDIDATE]: Term 1 election: Requested vote from peers edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621), 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801)
I20260620 06:04:40.058123 17584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d2f6420d24959b2a44d31117ec35f" candidate_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd17449f93d41349906e355b3c3a5be"
I20260620 06:04:40.058257 17584 raft_consensus.cc:3037] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:40.058148 17718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850d2f6420d24959b2a44d31117ec35f" candidate_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c38c30657914004b51cc15231259f02"
I20260620 06:04:40.058416 17718 raft_consensus.cc:3037] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:40.059439 17584 raft_consensus.cc:2454] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6120ff42da8c49c2bbf9aa7b6966183f in term 1.
I20260620 06:04:40.059548 17718 raft_consensus.cc:2454] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6120ff42da8c49c2bbf9aa7b6966183f in term 1.
I20260620 06:04:40.059898 17383 leader_election.cc:302] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6120ff42da8c49c2bbf9aa7b6966183f, edd17449f93d41349906e355b3c3a5be; no voters:
I20260620 06:04:40.060088 17787 raft_consensus.cc:2781] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:40.060318 17787 raft_consensus.cc:686] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 LEADER]: Becoming Leader. State: Replica: 6120ff42da8c49c2bbf9aa7b6966183f, State: Running, Role: LEADER
I20260620 06:04:40.060762 17787 consensus_queue.cc:237] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:40.064469 17304 catalog_manager.cc:5495] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1). New cstate: current_term: 1 leader_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } health_report { overall_health: UNKNOWN } } }
W20260620 06:04:40.065928 17765 tablet.cc:2369] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:04:40.079497 17497 tablet.cc:2369] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:04:40.089262 17631 tablet.cc:2369] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:04:40.110266 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.106309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49082 (local address 127.3.115.1:34795)
0620 06:04:40.106448 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:40.106452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:40.106465 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:04:40.106544 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:40.106546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:40.106605 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:40.106696 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:04:40.106966 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.107487 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.108550 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.108880 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.108971 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.109316 (+ 345us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:04:40.109832 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:04:40.109976 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:40.110040 (+ 64us) server_negotiation.cc:299] Negotiation successful
0620 06:04:40.110128 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:04:40.116703 17718 raft_consensus.cc:1261] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Refusing update from remote peer 6120ff42da8c49c2bbf9aa7b6966183f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:40.116703 17584 raft_consensus.cc:1261] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Refusing update from remote peer 6120ff42da8c49c2bbf9aa7b6966183f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:40.117630 17787 consensus_queue.cc:1035] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:40.117962 17792 consensus_queue.cc:1035] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:40.127285 17796 mvcc.cc:205] Tried to move back new op lower bound from 7298807726549381120 to 7298807726332059648. Current Snapshot: MvccSnapshot[applied={T|T < 7298807726549381120}]
I20260620 06:04:40.134697 17305 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:40.134909 17305 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:40.137745 17305 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Sending DeleteTablet for 3 replicas of tablet 850d2f6420d24959b2a44d31117ec35f
I20260620 06:04:40.138620 17430 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d2f6420d24959b2a44d31117ec35f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:40 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49070
I20260620 06:04:40.138778 17564 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d2f6420d24959b2a44d31117ec35f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:40 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:45102
I20260620 06:04:40.139242 17804 tablet_replica.cc:330] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: stopping tablet replica
I20260620 06:04:40.139235 17697 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d2f6420d24959b2a44d31117ec35f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:40 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:46782
I20260620 06:04:40.139616 17804 raft_consensus.cc:2229] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:04:40.139796 17806 tablet_replica.cc:330] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: stopping tablet replica
I20260620 06:04:40.139914 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17367
I20260620 06:04:40.140026 17804 raft_consensus.cc:2258] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:40.140267 17806 raft_consensus.cc:2229] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:04:40.140421 17805 tablet_replica.cc:330] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: stopping tablet replica
I20260620 06:04:40.140738 17805 raft_consensus.cc:2229] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:04:40.140750 17806 raft_consensus.cc:2258] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:40.141054 17805 raft_consensus.cc:2258] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:40.142499 17806 ts_tablet_manager.cc:1905] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:04:40.142688 17805 ts_tablet_manager.cc:1905] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:04:40.144955 17805 ts_tablet_manager.cc:1918] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:40.145074 17805 log.cc:1196] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/850d2f6420d24959b2a44d31117ec35f
I20260620 06:04:40.145351 17805 ts_tablet_manager.cc:1939] T 850d2f6420d24959b2a44d31117ec35f P edd17449f93d41349906e355b3c3a5be: Deleting consensus metadata
I20260620 06:04:40.146107 17806 ts_tablet_manager.cc:1918] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:40.146220 17806 log.cc:1196] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/850d2f6420d24959b2a44d31117ec35f
I20260620 06:04:40.146577 17806 ts_tablet_manager.cc:1939] T 850d2f6420d24959b2a44d31117ec35f P 5c38c30657914004b51cc15231259f02: Deleting consensus metadata
I20260620 06:04:40.146696 17289 catalog_manager.cc:4841] TS edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621): tablet 850d2f6420d24959b2a44d31117ec35f (table test-table [id=4164f15e91e8431897b7ec65b7b3e4cf]) successfully deleted
I20260620 06:04:40.147949 17289 catalog_manager.cc:4841] TS 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801): tablet 850d2f6420d24959b2a44d31117ec35f (table test-table [id=4164f15e91e8431897b7ec65b7b3e4cf]) successfully deleted
W20260620 06:04:40.148798 17291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:34795 (error 108)
I20260620 06:04:40.149626 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17501
I20260620 06:04:40.150009 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.149630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:34795 (local address 127.0.0.1:49084)
0620 06:04:40.149811 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:40.149898 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260620 06:04:40.150553 17291 catalog_manager.cc:4584] TS 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 850d2f6420d24959b2a44d31117ec35f: Network error: Client connection negotiation failed: client connection to 127.3.115.1:34795: connect: Connection refused (error 111)
I20260620 06:04:40.155725 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17635
I20260620 06:04:40.162132 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17273
I20260620 06:04:40.169147 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37891
--webserver_interface=127.3.115.62
--webserver_port=44785
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:40.288436 17808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:40.288930 17808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:40.289011 17808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:40.293313 17808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:40.293390 17808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:40.293409 17808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:40.293426 17808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:40.293478 17808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:40.299232 17808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=44785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:40.300452 17808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:40.301627 17808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:40.315696 17817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.315759 17815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.315701 17814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:40.316025 17808 server_base.cc:1034] running on GCE node
I20260620 06:04:40.316735 17808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:40.318086 17808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:40.319329 17808 hybrid_clock.cc:648] HybridClock initialized: now 1781935480319305 us; error 40 us; skew 500 ppm
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:40.322600 17808 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:40.323576 17808 webserver.cc:460] Webserver started at http://127.3.115.62:44785/ using document root <none> and password file <none>
I20260620 06:04:40.324173 17808 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:40.324259 17808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:40.328846 17808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:04:40.331339 17824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:40.332989 17808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:40.333117 17808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "8eb29e75041f4e678c044a155b9f5712"
format_stamp: "Formatted at 2026-06-20 06:04:38 on dist-test-slave-4h14"
server_key: "1cbbcbf082651d6e990d8708d99d40b3"
server_key_iv: "8e23b9bb94298c0a53ad23d631517f77"
server_key_version: "encryptionkey@0"
I20260620 06:04:40.333751 17808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:40.376545 17808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:40.380093 17808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:40.380263 17808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:40.388557 17808 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:37891
I20260620 06:04:40.388908 17876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:37891 every 8 connection(s)
I20260620 06:04:40.389824 17808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:40.394455 17877 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:04:40.396215 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap starting.
I20260620 06:04:40.397472 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17808
I20260620 06:04:40.398012 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:34795
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:37891
--webserver_port=46387
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:40.415242 17877 log.cc:824] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:40.424785 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:40.425197 17877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap complete.
I20260620 06:04:40.429533 17877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:40.429996 17877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb29e75041f4e678c044a155b9f5712, State: Initialized, Role: FOLLOWER
I20260620 06:04:40.430522 17877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:40.430677 17877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:40.430775 17877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:40.430903 17877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:04:40.433101 17877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:40.433480 17877 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb29e75041f4e678c044a155b9f5712; no voters:
I20260620 06:04:40.433846 17877 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:04:40.433934 17881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:04:40.434146 17881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 LEADER]: Becoming Leader. State: Replica: 8eb29e75041f4e678c044a155b9f5712, State: Running, Role: LEADER
I20260620 06:04:40.434453 17881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:40.435127 17877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:40.435878 17882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8eb29e75041f4e678c044a155b9f5712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } } }
I20260620 06:04:40.436005 17882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:40.436702 17883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb29e75041f4e678c044a155b9f5712. Latest consensus state: current_term: 2 leader_uuid: "8eb29e75041f4e678c044a155b9f5712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } } }
I20260620 06:04:40.436942 17883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:40.437067 17889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:40.442396 17889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:40.443203 17889 catalog_manager.cc:1259] Loaded cluster ID: 79b26b70ecbf4ecb96df586d1c243f21
I20260620 06:04:40.443275 17889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:40.444747 17889 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 06:04:40.450067 17889 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 06:04:40.456185 17889 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:40.457139 17889 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Loaded TSK: 0
I20260620 06:04:40.458431 17889 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:04:40.520737 17879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:40.521057 17879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:40.521152 17879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:40.525381 17879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:40.525470 17879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:40.525560 17879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:04:40.531093 17879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:34795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=46387
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:40.532437 17879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:40.534029 17879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:40.544930 17907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.545076 17905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.544939 17904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:40.545429 17879 server_base.cc:1034] running on GCE node
I20260620 06:04:40.545966 17879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:40.546808 17879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:40.548013 17879 hybrid_clock.cc:648] HybridClock initialized: now 1781935480547988 us; error 34 us; skew 500 ppm
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:40.551529 17879 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:04:40.552794 17879 webserver.cc:460] Webserver started at http://127.3.115.1:46387/ using document root <none> and password file <none>
I20260620 06:04:40.553511 17879 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:40.553622 17879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:40.559113 17879 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260620 06:04:40.562106 17914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:40.563800 17879 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:04:40.563957 17879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "7c9a49ba65ee8607e92bd6c27ab2027d"
server_key_iv: "35feea0cd61403db23ed32826e63090e"
server_key_version: "encryptionkey@0"
I20260620 06:04:40.564751 17879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:40.598914 17879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:40.602663 17879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:40.602941 17879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:40.603688 17879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:40.604924 17921 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:04:40.609500 17879 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:04:40.609586 17879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 06:04:40.609674 17879 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:04:40.611582 17879 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:04:40.611636 17879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:04:40.612063 17921 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap starting.
I20260620 06:04:40.623139 17879 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:34795
I20260620 06:04:40.623298 18028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:34795 every 8 connection(s)
I20260620 06:04:40.624297 17879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:04:40.625794 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 17879
I20260620 06:04:40.626261 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:33621
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:37891
--webserver_port=37701
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:40.631623 17921 log.cc:824] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:40.640498 17921 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:40.640991 17921 tablet_bootstrap.cc:492] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap complete.
I20260620 06:04:40.642539 17921 ts_tablet_manager.cc:1397] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.008s
I20260620 06:04:40.645691 17921 raft_consensus.cc:348] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:40.646265 17921 raft_consensus.cc:729] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6120ff42da8c49c2bbf9aa7b6966183f, State: Initialized, Role: FOLLOWER
I20260620 06:04:40.646354 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.626719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:58069)
0620 06:04:40.627998 (+ 1279us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:40.628045 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:40.628989 (+ 944us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:40.630214 (+ 1225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:40.630222 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:40.630690 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:40.631191 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.631207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.632967 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.632974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.634652 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.634673 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.635787 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.635793 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.636051 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.637003 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:40.637031 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:40.639349 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:40.642471 (+ 3122us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:40.642479 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:40.642494 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:40.642858 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:40.643325 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:40.643329 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:40.643331 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:40.643571 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:40.644354 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:40.644362 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:40.644715 (+ 353us) client_negotiation.cc:769] Sending connection context
0620 06:04:40.644927 (+ 212us) client_negotiation.cc:240] Negotiation successful
0620 06:04:40.645290 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"thread_start_us":156,"threads_started":1}
I20260620 06:04:40.646796 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.626726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:58069 (local address 127.3.115.62:37891)
0620 06:04:40.627507 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:40.627530 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:40.628187 (+ 657us) server_negotiation.cc:407] Connection header received
0620 06:04:40.629449 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:40.629461 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:40.629802 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:40.630154 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:40.631385 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.632782 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.634847 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.635608 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.635801 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.639627 (+ 3826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:40.639635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:40.639653 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:40.639693 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:40.642310 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:40.643001 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:40.643007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:40.643013 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:40.643203 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:40.643691 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:40.643695 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:40.643719 (+ 24us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:40.644222 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:40.644360 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:40.645077 (+ 717us) server_negotiation.cc:299] Negotiation successful
0620 06:04:40.645409 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":126,"threads_started":1}
I20260620 06:04:40.646809 17921 consensus_queue.cc:260] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } } peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } }
I20260620 06:04:40.647925 17921 ts_tablet_manager.cc:1428] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260620 06:04:40.649492 18029 heartbeater.cc:347] Connected to a master server at 127.3.115.62:37891
I20260620 06:04:40.649760 18029 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:40.650318 18029 heartbeater.cc:511] Master 127.3.115.62:37891 requested a full tablet report, sending...
I20260620 06:04:40.652972 17841 ts_manager.cc:194] Registered new tserver with Master: 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795)
I20260620 06:04:40.657039 17841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:58069
I20260620 06:04:40.659852 18029 heartbeater.cc:503] Master 127.3.115.62:37891 was elected leader, sending a full tablet report...
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:04:40.667665 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.656327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:34795 (local address 127.0.0.1:49086)
0620 06:04:40.657232 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:40.657288 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:40.657440 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:40.658006 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:40.658012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:40.658426 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:40.658536 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.658543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.659517 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.659524 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.660263 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.660274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.661078 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.661081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.661214 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.661893 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:40.661906 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:40.663483 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:40.665690 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:40.665702 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:40.665708 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:40.665979 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:40.666310 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:40.666314 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:40.666316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:40.666412 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:40.666923 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:40.666930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:40.667071 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 06:04:40.667216 (+ 145us) client_negotiation.cc:240] Negotiation successful
0620 06:04:40.667465 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":123,"threads_started":1}
I20260620 06:04:40.667665 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.656465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49086 (local address 127.3.115.1:34795)
0620 06:04:40.656969 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:40.656980 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:40.657406 (+ 426us) server_negotiation.cc:407] Connection header received
0620 06:04:40.657598 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:40.657603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:40.657828 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:40.657990 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:40.658679 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.659396 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.660396 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.660950 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.661060 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.663635 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:40.663646 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:40.663659 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:40.663687 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:40.665545 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:40.666114 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:40.666119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:40.666120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:40.666210 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:40.666515 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:40.666518 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:40.666520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:40.666803 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:40.666954 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:40.667314 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 06:04:40.667500 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":111,"threads_started":1}
I20260620 06:04:40.668998 17963 tablet_service.cc:1514] Processing DeleteTablet for tablet 850d2f6420d24959b2a44d31117ec35f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:40 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49086
I20260620 06:04:40.670017 18038 tablet_replica.cc:330] stopping tablet replica
I20260620 06:04:40.670409 18038 raft_consensus.cc:2229] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:04:40.670727 18038 raft_consensus.cc:2258] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:40.671944 18038 ts_tablet_manager.cc:1905] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:04:40.678018 18038 ts_tablet_manager.cc:1918] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:40.678167 18038 log.cc:1196] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/850d2f6420d24959b2a44d31117ec35f
I20260620 06:04:40.678488 18038 ts_tablet_manager.cc:1939] T 850d2f6420d24959b2a44d31117ec35f P 6120ff42da8c49c2bbf9aa7b6966183f: Deleting consensus metadata
I20260620 06:04:40.680953 17827 catalog_manager.cc:4841] TS 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795): tablet 850d2f6420d24959b2a44d31117ec35f (table test-table [id=4164f15e91e8431897b7ec65b7b3e4cf]) successfully deleted
W20260620 06:04:40.761883 18033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:40.762241 18033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:40.762321 18033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:40.766722 18033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:40.766826 18033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:40.766923 18033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:04:40.772775 18033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:33621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=37701
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:40.773983 18033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:40.775107 18033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:40.785532 18043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.785517 18046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.785509 18044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:40.785858 18033 server_base.cc:1034] running on GCE node
I20260620 06:04:40.786360 18033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:40.787111 18033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:40.788313 18033 hybrid_clock.cc:648] HybridClock initialized: now 1781935480788289 us; error 39 us; skew 500 ppm
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:40.791313 18033 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:04:40.792253 18033 webserver.cc:460] Webserver started at http://127.3.115.2:37701/ using document root <none> and password file <none>
I20260620 06:04:40.792894 18033 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:40.793011 18033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:40.797163 18033 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:04:40.799397 18053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:40.800873 18033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:40.801011 18033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "edd17449f93d41349906e355b3c3a5be"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "2639a29820c70481723a98cb26af3acf"
server_key_iv: "4c30c1796ac6a858fce05fdfe28a25e1"
server_key_version: "encryptionkey@0"
I20260620 06:04:40.801626 18033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:40.818902 18033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:40.822383 18033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:40.822608 18033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:40.823266 18033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:40.824147 18033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:40.824220 18033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:40.824287 18033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:40.824330 18033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:40.833747 18033 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:33621
I20260620 06:04:40.833765 18166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:33621 every 8 connection(s)
I20260620 06:04:40.834939 18033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:04:40.837623 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18033
I20260620 06:04:40.838150 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:44801
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:37891
--webserver_port=37427
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:40.851678 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.837051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:38205 (local address 127.3.115.62:37891)
0620 06:04:40.837196 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:40.837200 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:40.839031 (+ 1831us) server_negotiation.cc:407] Connection header received
0620 06:04:40.840355 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:40.840361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:40.840434 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:40.840543 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:40.841933 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.842858 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.844069 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.844530 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.844706 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.847528 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:40.847531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:40.847533 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:40.847563 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:40.849226 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:40.850005 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:40.850008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:40.850009 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:40.850090 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:40.850479 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:40.850481 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:40.850482 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:40.850664 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:40.850741 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:40.851420 (+ 679us) server_negotiation.cc:299] Negotiation successful
0620 06:04:40.851547 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:04:40.852887 18169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:40.837639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:38205)
0620 06:04:40.838861 (+ 1222us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:40.838918 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:40.840116 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:40.840779 (+ 663us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:40.840786 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:40.841166 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:40.841725 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.841747 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.842995 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.843003 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.843935 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:40.843944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:40.844759 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:40.844763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:40.844918 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:40.845551 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:40.845569 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:40.847310 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:40.849370 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:40.849380 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:40.849439 (+ 59us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:40.849892 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:40.850182 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:40.850185 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:40.850187 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:40.850388 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:40.850770 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:40.850778 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:40.851137 (+ 359us) client_negotiation.cc:769] Sending connection context
0620 06:04:40.851355 (+ 218us) client_negotiation.cc:240] Negotiation successful
0620 06:04:40.851749 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":977,"thread_start_us":775,"threads_started":1}
I20260620 06:04:40.854281 18167 heartbeater.cc:347] Connected to a master server at 127.3.115.62:37891
I20260620 06:04:40.854628 18167 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:40.855199 18167 heartbeater.cc:511] Master 127.3.115.62:37891 requested a full tablet report, sending...
I20260620 06:04:40.856438 17841 ts_manager.cc:194] Registered new tserver with Master: edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621)
I20260620 06:04:40.857116 17841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:38205
W20260620 06:04:40.959986 18170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:40.960321 18170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:40.960371 18170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:40.964843 18170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:40.964918 18170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:40.965014 18170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:04:40.970755 18170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:44801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=37427
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:40.972031 18170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:40.973321 18170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:40.985357 18178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.985172 18175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:40.985461 18176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:40.985853 18170 server_base.cc:1034] running on GCE node
I20260620 06:04:40.986358 18170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:40.987094 18170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:40.988312 18170 hybrid_clock.cc:648] HybridClock initialized: now 1781935480988286 us; error 41 us; skew 500 ppm
Jun 20 06:04:40 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:40.991705 18170 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:04:40.992796 18170 webserver.cc:460] Webserver started at http://127.3.115.3:37427/ using document root <none> and password file <none>
I20260620 06:04:40.993382 18170 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:40.993458 18170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:40.997893 18170 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:04:41.000311 18185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:41.001868 18170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:04:41.001987 18170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "5c38c30657914004b51cc15231259f02"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "14e222b40e38f26d8c8f7a7dcc171ce2"
server_key_iv: "e8628ea3c7803e4b9180fb765c879b5f"
server_key_version: "encryptionkey@0"
I20260620 06:04:41.002589 18170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:41.024390 18170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:41.028095 18170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:41.028272 18170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:41.028939 18170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:41.029783 18170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:41.029831 18170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:41.029872 18170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:41.029888 18170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:41.040247 18170 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:44801
I20260620 06:04:41.040267 18298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:44801 every 8 connection(s)
I20260620 06:04:41.041484 18170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:04:41.046618 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18170
Jun 20 06:04:41 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935480, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:41.057111 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.047327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50046 (local address 127.3.115.62:37891)
0620 06:04:41.047562 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.047567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.047774 (+ 207us) server_negotiation.cc:407] Connection header received
0620 06:04:41.047908 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.047910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.047952 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.048021 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:41.048952 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.049721 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.051229 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.051592 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.051699 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.053269 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:41.053274 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:41.053277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:41.053308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:41.055131 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:41.055712 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:41.055717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:41.055722 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:41.055812 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:41.056144 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:41.056147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:41.056149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:41.056384 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:41.056560 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.056796 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.056932 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":61,"threads_started":1}
I20260620 06:04:41.057802 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.043456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:45753 (local address 127.3.115.62:37891)
0620 06:04:41.043612 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.043616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.044504 (+ 888us) server_negotiation.cc:407] Connection header received
0620 06:04:41.045874 (+ 1370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.045877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.045925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.046003 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:41.047471 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.048258 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.049923 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.050266 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.050382 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.053602 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:41.053605 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:41.053606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:41.053641 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:41.055238 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:41.055963 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:41.055967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:41.055969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:41.056047 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:41.056447 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:41.056452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:41.056454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:41.056651 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:41.056789 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.057481 (+ 692us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.057602 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:04:41.059159 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.043891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:45753)
0620 06:04:41.044335 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.044380 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:41.045614 (+ 1234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:41.046231 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:41.046238 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:41.046626 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:41.047308 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.047326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.048397 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.048400 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.049760 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.049773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.050438 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.050441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.050597 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.051283 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:41.051305 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:41.053407 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:41.055368 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:41.055373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:41.055387 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:41.055850 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:41.056144 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:41.056147 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:41.056149 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:41.056328 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:41.056791 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:41.056798 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:41.057191 (+ 393us) client_negotiation.cc:769] Sending connection context
0620 06:04:41.057541 (+ 350us) client_negotiation.cc:240] Negotiation successful
0620 06:04:41.057951 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
I20260620 06:04:41.060341 18299 heartbeater.cc:347] Connected to a master server at 127.3.115.62:37891
I20260620 06:04:41.060578 18299 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:41.061122 18299 heartbeater.cc:511] Master 127.3.115.62:37891 requested a full tablet report, sending...
I20260620 06:04:41.062196 17841 ts_manager.cc:194] Registered new tserver with Master: 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801)
I20260620 06:04:41.062805 17841 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:45753
I20260620 06:04:41.071396 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:04:41.076254 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.072155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50054 (local address 127.3.115.62:37891)
0620 06:04:41.072327 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.072333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.072498 (+ 165us) server_negotiation.cc:407] Connection header received
0620 06:04:41.072750 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.072752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.072864 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.072979 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:04:41.073245 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.073777 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.074728 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.074961 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.075074 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.075413 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:04:41.075851 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:04:41.075995 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.076032 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.076088 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260620 06:04:41.077243 17841 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:04:41.080598 17841 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:04:41.089483 17963 tablet_service.cc:1467] Processing CreateTablet for tablet fad0f812f3984ac489c446e22798efd7 (DEFAULT_TABLE table=test-table [id=93e0157a09274af480d7aa48f54fb543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:41.090250 17963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad0f812f3984ac489c446e22798efd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:41.093405 17921 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap starting.
I20260620 06:04:41.094278 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.088533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:44801 (local address 127.0.0.1:46786)
0620 06:04:41.088889 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.088911 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:41.088999 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:41.089860 (+ 861us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:41.089863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:41.089881 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:41.090006 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.090012 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.091115 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.091118 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.092378 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.092390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.093809 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.093813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.093911 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.093959 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 06:04:41.094044 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:04:41.094130 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":152,"threads_started":1}
I20260620 06:04:41.094329 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.088962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.3.115.3:44801)
0620 06:04:41.089243 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.089255 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.089284 (+ 29us) server_negotiation.cc:407] Connection header received
0620 06:04:41.089352 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.089375 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.089625 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.089869 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:41.090121 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.090976 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.092528 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.093663 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.093796 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.093957 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.094085 (+ 128us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.094184 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":106,"threads_started":1}
I20260620 06:04:41.094417 17921 tablet_bootstrap.cc:654] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:41.096349 17921 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: No bootstrap required, opened a new log
I20260620 06:04:41.096455 17921 ts_tablet_manager.cc:1397] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:04:41.096959 17921 raft_consensus.cc:348] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.097097 17921 raft_consensus.cc:374] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:41.097137 17921 raft_consensus.cc:729] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6120ff42da8c49c2bbf9aa7b6966183f, State: Initialized, Role: FOLLOWER
I20260620 06:04:41.097301 17921 consensus_queue.cc:260] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.098039 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.088296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33621 (local address 127.0.0.1:45118)
0620 06:04:41.088442 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.088458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:41.088549 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:41.093564 (+ 5015us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:41.093567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:41.093588 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:41.093685 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.093692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.095595 (+ 1903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.095599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.096541 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.096555 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.097610 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.097613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.097714 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.097737 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:04:41.097809 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:04:41.097899 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:04:41.098058 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.088519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45118 (local address 127.3.115.2:33621)
0620 06:04:41.092809 (+ 4290us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.092917 (+ 108us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.092952 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:04:41.093053 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.093061 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.093324 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.093869 (+ 545us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:41.094041 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.095390 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.096709 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.097432 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.097604 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.097779 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.097836 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.097923 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":156,"threads_started":1}
I20260620 06:04:41.098217 18232 tablet_service.cc:1467] Processing CreateTablet for tablet fad0f812f3984ac489c446e22798efd7 (DEFAULT_TABLE table=test-table [id=93e0157a09274af480d7aa48f54fb543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:41.099334 17921 ts_tablet_manager.cc:1428] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260620 06:04:41.099540 18232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad0f812f3984ac489c446e22798efd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:41.101271 18101 tablet_service.cc:1467] Processing CreateTablet for tablet fad0f812f3984ac489c446e22798efd7 (DEFAULT_TABLE table=test-table [id=93e0157a09274af480d7aa48f54fb543]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:41.102283 18101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad0f812f3984ac489c446e22798efd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:41.107997 18312 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Bootstrap starting.
I20260620 06:04:41.110405 18313 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Bootstrap starting.
I20260620 06:04:41.111013 18312 tablet_bootstrap.cc:654] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:41.112097 18312 log.cc:824] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:41.113057 18313 tablet_bootstrap.cc:654] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:41.113947 18313 log.cc:824] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:41.114506 18312 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: No bootstrap required, opened a new log
I20260620 06:04:41.114733 18312 ts_tablet_manager.cc:1397] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 06:04:41.116094 18313 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: No bootstrap required, opened a new log
I20260620 06:04:41.116297 18313 ts_tablet_manager.cc:1397] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:04:41.117952 18312 raft_consensus.cc:348] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.118248 18312 raft_consensus.cc:374] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:41.118333 18312 raft_consensus.cc:729] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c38c30657914004b51cc15231259f02, State: Initialized, Role: FOLLOWER
I20260620 06:04:41.118785 18312 consensus_queue.cc:260] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.119513 18299 heartbeater.cc:503] Master 127.3.115.62:37891 was elected leader, sending a full tablet report...
I20260620 06:04:41.119840 18312 ts_tablet_manager.cc:1428] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:04:41.119715 18313 raft_consensus.cc:348] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.120007 18313 raft_consensus.cc:374] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:41.120064 18313 raft_consensus.cc:729] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd17449f93d41349906e355b3c3a5be, State: Initialized, Role: FOLLOWER
I20260620 06:04:41.120513 18313 consensus_queue.cc:260] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.121269 18167 heartbeater.cc:503] Master 127.3.115.62:37891 was elected leader, sending a full tablet report...
I20260620 06:04:41.121711 18313 ts_tablet_manager.cc:1428] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260620 06:04:41.125695 18030 tablet.cc:2369] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:04:41.204291 18316 raft_consensus.cc:482] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:41.204587 18316 raft_consensus.cc:504] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.206211 18316 leader_election.cc:288] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621), 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795)
I20260620 06:04:41.210230 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.206452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33621 (local address 127.3.115.3:47663)
0620 06:04:41.206596 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.206616 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:41.206725 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:41.207048 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:41.207051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:41.207078 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:41.207212 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.207218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.208031 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.208038 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.208897 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.208906 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.209554 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.209558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.209679 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.209715 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:04:41.209810 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:04:41.209992 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":1,"spinlock_wait_cycles":1920}
I20260620 06:04:41.210294 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.206578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:47663 (local address 127.3.115.2:33621)
0620 06:04:41.206704 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.206710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.206738 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:04:41.206859 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.206867 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.206945 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.207040 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:41.207327 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.207919 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.209032 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.209434 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.209551 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.209652 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.209835 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.209996 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:04:41.211076 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.206546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:34795 (local address 127.3.115.3:51993)
0620 06:04:41.206949 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.206964 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:41.207069 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:41.207369 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:41.207372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:41.207388 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:41.207507 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.207512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.208278 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.208285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.209139 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:41.209148 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.210731 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.210734 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:41.210823 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.210842 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:04:41.210918 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 06:04:41.210972 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"mutex_wait_us":10,"thread_start_us":126,"threads_started":1}
I20260620 06:04:41.211027 18121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad0f812f3984ac489c446e22798efd7" candidate_uuid: "5c38c30657914004b51cc15231259f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd17449f93d41349906e355b3c3a5be" is_pre_election: true
I20260620 06:04:41.211174 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.206817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:51993 (local address 127.3.115.1:34795)
0620 06:04:41.207053 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.207058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.207072 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:04:41.207161 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.207164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.207234 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.207350 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:41.207627 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.208158 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.209309 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.210222 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.210479 (+ 257us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.210689 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.210953 (+ 264us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.211041 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":67,"threads_started":1}
I20260620 06:04:41.211418 18121 raft_consensus.cc:2454] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c38c30657914004b51cc15231259f02 in term 0.
I20260620 06:04:41.211769 17983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad0f812f3984ac489c446e22798efd7" candidate_uuid: "5c38c30657914004b51cc15231259f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" is_pre_election: true
I20260620 06:04:41.211990 18186 leader_election.cc:302] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c38c30657914004b51cc15231259f02, edd17449f93d41349906e355b3c3a5be; no voters:
I20260620 06:04:41.212088 17983 raft_consensus.cc:2454] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c38c30657914004b51cc15231259f02 in term 0.
I20260620 06:04:41.212304 18316 raft_consensus.cc:2781] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:04:41.212411 18316 raft_consensus.cc:482] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:04:41.212476 18316 raft_consensus.cc:3037] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:41.213758 18316 raft_consensus.cc:504] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.214180 18316 leader_election.cc:288] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [CANDIDATE]: Term 1 election: Requested vote from peers edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621), 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795)
I20260620 06:04:41.214522 17983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad0f812f3984ac489c446e22798efd7" candidate_uuid: "5c38c30657914004b51cc15231259f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
I20260620 06:04:41.214545 18121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad0f812f3984ac489c446e22798efd7" candidate_uuid: "5c38c30657914004b51cc15231259f02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd17449f93d41349906e355b3c3a5be"
I20260620 06:04:41.214679 17983 raft_consensus.cc:3037] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:41.214679 18121 raft_consensus.cc:3037] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:41.215739 17983 raft_consensus.cc:2454] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c38c30657914004b51cc15231259f02 in term 1.
I20260620 06:04:41.215806 18121 raft_consensus.cc:2454] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c38c30657914004b51cc15231259f02 in term 1.
I20260620 06:04:41.216063 18188 leader_election.cc:302] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c38c30657914004b51cc15231259f02, 6120ff42da8c49c2bbf9aa7b6966183f; no voters:
I20260620 06:04:41.216305 18316 raft_consensus.cc:2781] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:41.216487 18316 raft_consensus.cc:686] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 1 LEADER]: Becoming Leader. State: Replica: 5c38c30657914004b51cc15231259f02, State: Running, Role: LEADER
I20260620 06:04:41.216868 18316 consensus_queue.cc:237] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.220170 17841 catalog_manager.cc:5495] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c38c30657914004b51cc15231259f02 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "5c38c30657914004b51cc15231259f02" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } health_report { overall_health: UNKNOWN } } }
I20260620 06:04:41.253329 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.249441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.3.115.3:44801)
0620 06:04:41.249584 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:41.249587 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:41.249603 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:04:41.249684 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:41.249686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:41.249752 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:41.249866 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:04:41.250156 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.250699 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.251620 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:41.251912 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:41.252062 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:41.252423 (+ 361us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:04:41.252901 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:04:41.253049 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:41.253111 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 06:04:41.253176 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 06:04:41.259843 17983 raft_consensus.cc:1261] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Refusing update from remote peer 5c38c30657914004b51cc15231259f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:41.259846 18121 raft_consensus.cc:1261] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Refusing update from remote peer 5c38c30657914004b51cc15231259f02: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:41.260622 18316 consensus_queue.cc:1035] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:41.260946 18320 consensus_queue.cc:1035] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:41.266716 18323 mvcc.cc:205] Tried to move back new op lower bound from 7298807731232464896 to 7298807731067244544. Current Snapshot: MvccSnapshot[applied={T|T < 7298807731232464896}]
I20260620 06:04:41.268977 18324 mvcc.cc:205] Tried to move back new op lower bound from 7298807731232464896 to 7298807731067244544. Current Snapshot: MvccSnapshot[applied={T|T < 7298807731232464896}]
I20260620 06:04:41.269163 18325 mvcc.cc:205] Tried to move back new op lower bound from 7298807731232464896 to 7298807731067244544. Current Snapshot: MvccSnapshot[applied={T|T < 7298807731232464896}]
I20260620 06:04:41.277875 17841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:41.278112 17841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:41.280836 17841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Sending DeleteTablet for 3 replicas of tablet fad0f812f3984ac489c446e22798efd7
I20260620 06:04:41.281838 17963 tablet_service.cc:1514] Processing DeleteTablet for tablet fad0f812f3984ac489c446e22798efd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:49086
I20260620 06:04:41.282012 18232 tablet_service.cc:1514] Processing DeleteTablet for tablet fad0f812f3984ac489c446e22798efd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:46786
I20260620 06:04:41.282325 18101 tablet_service.cc:1514] Processing DeleteTablet for tablet fad0f812f3984ac489c446e22798efd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:41 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:45118
I20260620 06:04:41.283006 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17879
I20260620 06:04:41.283226 18334 tablet_replica.cc:330] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: stopping tablet replica
I20260620 06:04:41.284230 18334 raft_consensus.cc:2229] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:04:41.285112 18334 raft_consensus.cc:2258] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:41.288158 18334 ts_tablet_manager.cc:1905] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 06:04:41.291482 17827 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:34795 (error 108)
I20260620 06:04:41.292450 18333 tablet_replica.cc:330] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: stopping tablet replica
I20260620 06:04:41.292865 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 18033
I20260620 06:04:41.292852 18334 ts_tablet_manager.cc:1918] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:41.293115 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.292780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:34795 (local address 127.0.0.1:49100)
0620 06:04:41.292967 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.293031 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:34795: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260620 06:04:41.293570 17827 catalog_manager.cc:4584] TS 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fad0f812f3984ac489c446e22798efd7: Network error: Client connection negotiation failed: client connection to 127.3.115.1:34795: connect: Connection refused (error 111)
I20260620 06:04:41.294872 18334 log.cc:1196] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/fad0f812f3984ac489c446e22798efd7
I20260620 06:04:41.295280 18334 ts_tablet_manager.cc:1939] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Deleting consensus metadata
I20260620 06:04:41.296815 17825 catalog_manager.cc:4841] TS 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801): tablet fad0f812f3984ac489c446e22798efd7 (table test-table [id=93e0157a09274af480d7aa48f54fb543]) successfully deleted
I20260620 06:04:41.299757 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 18170
I20260620 06:04:41.300987 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.300584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33621 (local address 127.0.0.1:45126)
0620 06:04:41.300805 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.300873 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.2:33621: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260620 06:04:41.306020 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 17808
I20260620 06:04:41.312973 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37891
--webserver_interface=127.3.115.62
--webserver_port=44785
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:37891
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:41.432762 18336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:41.433121 18336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:41.433205 18336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:41.437506 18336 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:41.437630 18336 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:41.437670 18336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:41.437712 18336 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:41.437772 18336 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:41.443745 18336 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:37891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=44785
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18336
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:41.444996 18336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:41.446139 18336 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:41.455710 18344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:41.455710 18342 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:41.455770 18341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:41.456148 18336 server_base.cc:1034] running on GCE node
I20260620 06:04:41.456908 18336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:41.458192 18336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:41.459408 18336 hybrid_clock.cc:648] HybridClock initialized: now 1781935481459368 us; error 61 us; skew 500 ppm
Jun 20 06:04:41 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:41.462405 18336 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:41.463413 18336 webserver.cc:460] Webserver started at http://127.3.115.62:44785/ using document root <none> and password file <none>
I20260620 06:04:41.464026 18336 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:41.464125 18336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:41.468557 18336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:41.471130 18351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:41.472612 18336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:04:41.472955 18336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "8eb29e75041f4e678c044a155b9f5712"
format_stamp: "Formatted at 2026-06-20 06:04:38 on dist-test-slave-4h14"
server_key: "1cbbcbf082651d6e990d8708d99d40b3"
server_key_iv: "8e23b9bb94298c0a53ad23d631517f77"
server_key_version: "encryptionkey@0"
I20260620 06:04:41.473579 18336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:41.486912 18336 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:41.490782 18336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:41.491040 18336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:41.498951 18336 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:37891
I20260620 06:04:41.499289 18403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:37891 every 8 connection(s)
I20260620 06:04:41.500140 18336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:41.500877 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18336
I20260620 06:04:41.501391 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:34795
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:37891
--webserver_port=46387
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:41.504969 18404 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:04:41.506871 18404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap starting.
I20260620 06:04:41.525115 18404 log.cc:824] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:41.537202 18404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:41.537705 18404 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712: Bootstrap complete.
I20260620 06:04:41.542027 18404 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:41.542490 18404 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb29e75041f4e678c044a155b9f5712, State: Initialized, Role: FOLLOWER
I20260620 06:04:41.543033 18404 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:41.543188 18404 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:41.543282 18404 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:41.543411 18404 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:04:41.545660 18404 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:41.546097 18404 leader_election.cc:302] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8eb29e75041f4e678c044a155b9f5712; no voters:
I20260620 06:04:41.546489 18404 leader_election.cc:288] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 06:04:41.546589 18408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:04:41.546792 18408 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [term 3 LEADER]: Becoming Leader. State: Replica: 8eb29e75041f4e678c044a155b9f5712, State: Running, Role: LEADER
I20260620 06:04:41.547098 18408 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } }
I20260620 06:04:41.547816 18404 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:41.548465 18409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8eb29e75041f4e678c044a155b9f5712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } } }
I20260620 06:04:41.548594 18409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:41.548843 18409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8eb29e75041f4e678c044a155b9f5712. Latest consensus state: current_term: 3 leader_uuid: "8eb29e75041f4e678c044a155b9f5712" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8eb29e75041f4e678c044a155b9f5712" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37891 } } }
I20260620 06:04:41.548945 18409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8eb29e75041f4e678c044a155b9f5712 [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:41.549778 18413 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:41.555145 18413 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:41.555927 18413 catalog_manager.cc:1259] Loaded cluster ID: 79b26b70ecbf4ecb96df586d1c243f21
I20260620 06:04:41.556003 18413 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 06:04:41.557101 18413 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa26a7f79ad google::LogMessage::Fail() at ??:0
@ 0x7fa26a7fa88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa26a7f74de google::LogMessage::Flush() at ??:0
@ 0x7fa26a7f8a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa275136fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fa275137d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa275134579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fa275178640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa26ca493d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa26ca49c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fa26ca4b3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fa26ca3a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa26e07e6db start_thread at ??:0
@ 0x7fa2692f561f clone at ??:0
W20260620 06:04:41.652578 18406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:41.652978 18406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:41.653026 18406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:41.657277 18406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:41.657359 18406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:41.657428 18406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:04:41.663148 18406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:34795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=46387
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:41.664366 18406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:41.665568 18406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:41.677112 18445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:41.677129 18446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:41.677112 18448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:41.677593 18406 server_base.cc:1034] running on GCE node
I20260620 06:04:41.678251 18406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:41.679057 18406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:41.680281 18406 hybrid_clock.cc:648] HybridClock initialized: now 1781935481680245 us; error 75 us; skew 500 ppm
Jun 20 06:04:41 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:41.683822 18406 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:04:41.685050 18406 webserver.cc:460] Webserver started at http://127.3.115.1:46387/ using document root <none> and password file <none>
I20260620 06:04:41.685707 18406 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:41.685813 18406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:41.690546 18406 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:04:41.693511 18455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:41.695101 18406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:41.695267 18406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "7c9a49ba65ee8607e92bd6c27ab2027d"
server_key_iv: "35feea0cd61403db23ed32826e63090e"
server_key_version: "encryptionkey@0"
I20260620 06:04:41.695925 18406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:41.723868 18406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:41.727648 18406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:41.727875 18406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:41.728530 18406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:41.729849 18462 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:04:41.734683 18406 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:04:41.734787 18406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 06:04:41.734874 18406 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:04:41.736801 18406 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:04:41.736881 18406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260620 06:04:41.737155 18462 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap starting.
I20260620 06:04:41.748689 18406 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:34795
I20260620 06:04:41.750149 18406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:04:41.750936 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:34795 every 8 connection(s)
I20260620 06:04:41.759981 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18406
I20260620 06:04:41.760540 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:33621
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:37891
--webserver_port=37701
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:41.765656 18462 log.cc:824] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:41.779578 18462 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:41.780066 18462 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Bootstrap complete.
I20260620 06:04:41.781616 18462 ts_tablet_manager.cc:1397] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent bootstrapping tablet: real 0.045s user 0.026s sys 0.009s
I20260620 06:04:41.785586 18462 raft_consensus.cc:348] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.786204 18462 raft_consensus.cc:729] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6120ff42da8c49c2bbf9aa7b6966183f, State: Initialized, Role: FOLLOWER
I20260620 06:04:41.786805 18462 consensus_queue.cc:260] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:41.788013 18462 ts_tablet_manager.cc:1428] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 06:04:41.818490 18572 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:04:41.755902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:32891)
0620 06:04:41.756554 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.756616 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:41.757893 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:41.817681 (+ 59788us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":212,"threads_started":1}
W20260620 06:04:41.819015 18570 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:37891 (0 consecutive failures): Network error: Failed to ping master at 127.3.115.62:37891: Client connection negotiation failed: client connection to 127.3.115.62:37891: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:04:41.819880 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.819536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:33855)
0620 06:04:41.819700 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.819796 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:04:41.820705 18572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.820437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:40699)
0620 06:04:41.820561 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.820618 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:04:41.820971 18570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:04:41.900949 18573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:41.901326 18573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:41.901432 18573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:41.905704 18573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:41.905812 18573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:41.905910 18573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:04:41.912297 18573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:33621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=37701
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:41.913565 18573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:41.914675 18573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:41.925347 18581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:41.925339 18580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:41.925631 18583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:41.925865 18573 server_base.cc:1034] running on GCE node
I20260620 06:04:41.926306 18573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:41.926975 18573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:41.928143 18573 hybrid_clock.cc:648] HybridClock initialized: now 1781935481928113 us; error 45 us; skew 500 ppm
Jun 20 06:04:41 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:41.931308 18573 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:04:41.932406 18573 webserver.cc:460] Webserver started at http://127.3.115.2:37701/ using document root <none> and password file <none>
I20260620 06:04:41.933080 18573 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:41.933182 18573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:41.937357 18573 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:04:41.939785 18590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:41.941430 18573 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:41.941597 18573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "edd17449f93d41349906e355b3c3a5be"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "2639a29820c70481723a98cb26af3acf"
server_key_iv: "4c30c1796ac6a858fce05fdfe28a25e1"
server_key_version: "encryptionkey@0"
I20260620 06:04:41.942211 18573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:41.953500 18573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:41.957109 18573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:41.957329 18573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:41.958045 18573 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:41.959388 18597 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:04:41.963936 18573 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:04:41.964049 18573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 06:04:41.964141 18573 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:04:41.966095 18573 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:04:41.966172 18573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260620 06:04:41.966596 18597 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Bootstrap starting.
I20260620 06:04:41.977049 18573 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:33621
I20260620 06:04:41.977126 18704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:33621 every 8 connection(s)
I20260620 06:04:41.978103 18573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:04:41.979422 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18573
I20260620 06:04:41.979934 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:44801
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:37891
--webserver_port=37427
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:40879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:04:41.982597 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.980533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:58425)
0620 06:04:41.981274 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.981456 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":172,"threads_started":1}
W20260620 06:04:41.983093 18705 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:37891 (0 consecutive failures): Network error: Failed to ping master at 127.3.115.62:37891: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
I20260620 06:04:41.983724 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.983483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:38939)
0620 06:04:41.983605 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.983660 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:04:41.984452 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:41.984237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:57843)
0620 06:04:41.984348 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:41.984393 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260620 06:04:41.984681 18705 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:04:41.988266 18597 log.cc:824] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:41.998564 18597 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:41.999061 18597 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Bootstrap complete.
I20260620 06:04:42.000680 18597 ts_tablet_manager.cc:1397] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Time spent bootstrapping tablet: real 0.034s user 0.030s sys 0.004s
I20260620 06:04:42.004778 18597 raft_consensus.cc:348] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:42.005393 18597 raft_consensus.cc:729] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd17449f93d41349906e355b3c3a5be, State: Initialized, Role: FOLLOWER
I20260620 06:04:42.006016 18597 consensus_queue.cc:260] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:42.007166 18597 ts_tablet_manager.cc:1428] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:04:42.007481 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:42.007147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:43871)
0620 06:04:42.007319 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:42.007403 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 06:04:42.008255 18707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:42.008015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:41545)
0620 06:04:42.008135 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:42.008183 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 06:04:42.103147 18708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:42.103430 18708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:42.103475 18708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:42.107595 18708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:42.107661 18708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:42.107764 18708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:04:42.113209 18708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:40879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:44801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=37427
--tserver_master_addrs=127.3.115.62:37891
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:42.114333 18708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:42.115401 18708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:42.125320 18716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:42.125394 18708 server_base.cc:1034] running on GCE node
W20260620 06:04:42.125346 18718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:42.125332 18715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:42.126087 18708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:42.126773 18708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:42.127949 18708 hybrid_clock.cc:648] HybridClock initialized: now 1781935482127930 us; error 31 us; skew 500 ppm
Jun 20 06:04:42 dist-test-slave-4h14 krb5kdc[17260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935482, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:42.130801 18708 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:04:42.131690 18708 webserver.cc:460] Webserver started at http://127.3.115.3:37427/ using document root <none> and password file <none>
I20260620 06:04:42.132270 18708 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:42.132369 18708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:42.136317 18708 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:04:42.138518 18725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:42.139885 18708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260620 06:04:42.139990 18708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "5c38c30657914004b51cc15231259f02"
format_stamp: "Formatted at 2026-06-20 06:04:39 on dist-test-slave-4h14"
server_key: "14e222b40e38f26d8c8f7a7dcc171ce2"
server_key_iv: "e8628ea3c7803e4b9180fb765c879b5f"
server_key_version: "encryptionkey@0"
I20260620 06:04:42.140512 18708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:42.172724 18708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:42.176412 18708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:42.176580 18708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:42.177244 18708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:42.178076 18708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:42.178123 18708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:42.178164 18708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:42.178180 18708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:42.187636 18708 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:44801
I20260620 06:04:42.187669 18838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:44801 every 8 connection(s)
I20260620 06:04:42.188796 18708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:04:42.192991 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:42.191101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:40249)
0620 06:04:42.191603 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:42.192191 (+ 588us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":119,"threads_started":1}
W20260620 06:04:42.193504 18839 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:37891 (0 consecutive failures): Network error: Failed to ping master at 127.3.115.62:37891: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
I20260620 06:04:42.194167 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:42.193954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:56421)
0620 06:04:42.194062 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:42.194112 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 06:04:42.194864 18841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:42.194579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:49635)
0620 06:04:42.194725 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:42.194773 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260620 06:04:42.195072 18839 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:04:42.199059 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18708
I20260620 06:04:42.822039 18843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:42.821625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:41337)
0620 06:04:42.821868 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:42.821925 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":73,"threads_started":1}
I20260620 06:04:43.009511 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.009052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:47965)
0620 06:04:43.009320 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.009376 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":71,"threads_started":1}
I20260620 06:04:43.196242 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.195787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:41847)
0620 06:04:43.196070 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.196132 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
I20260620 06:04:43.331861 18846 raft_consensus.cc:482] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:43.332126 18846 raft_consensus.cc:504] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:43.333395 18846 leader_election.cc:288] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801), 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795)
Jun 20 06:04:43 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.3@KRBTEST.COM
Jun 20 06:04:43 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935481, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:04:43.348696 18849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.333975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:44147 (local address 127.3.115.1:34795)
0620 06:04:43.334436 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:43.334444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:43.334471 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:04:43.334776 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:43.334780 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:43.334987 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:43.335179 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:43.336387 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.337414 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.338642 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.339145 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.339313 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:43.342759 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:43.342766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:43.342784 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:43.342821 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:43.344803 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:43.345647 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:43.345654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:43.345660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:43.345868 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:43.346491 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:43.346494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:43.346496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:43.347121 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:43.347273 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:43.348172 (+ 899us) server_negotiation.cc:299] Negotiation successful
0620 06:04:43.348476 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":105,"threads_started":1}
I20260620 06:04:43.348709 18848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.333869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:45443 (local address 127.3.115.3:44801)
0620 06:04:43.334340 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:43.334363 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:43.334410 (+ 47us) server_negotiation.cc:407] Connection header received
0620 06:04:43.335316 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:43.335321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:43.335605 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:43.335891 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:43.336349 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.337452 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.338657 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.339296 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.340044 (+ 748us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:43.342479 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:43.342484 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:43.342496 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:43.342526 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:43.344812 (+ 2286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:43.345647 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:43.345654 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:43.345660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:43.345970 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:43.346491 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:43.346494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:43.346496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:43.347095 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:43.347257 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:43.348172 (+ 915us) server_negotiation.cc:299] Negotiation successful
0620 06:04:43.348476 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":130,"threads_started":1}
I20260620 06:04:43.348744 18847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.333668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:34795 (local address 127.3.115.2:44147)
0620 06:04:43.334033 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.334048 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:43.334433 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:43.335265 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:43.335275 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:43.335756 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:43.336210 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:43.336224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.337558 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.337562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:43.338520 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:43.338529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.339284 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.339288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:43.339536 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:43.340380 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:43.340398 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:43.342587 (+ 2189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:43.344962 (+ 2375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:43.344969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:43.344986 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:43.345515 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:43.346077 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:43.346081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:43.346084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:43.346350 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:43.347242 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:43.347251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:43.347724 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 06:04:43.347988 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 06:04:43.348486 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":208,"spinlock_wait_cycles":33664,"thread_start_us":64,"threads_started":1}
I20260620 06:04:43.348745 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.333668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:44801 (local address 127.3.115.2:45443)
0620 06:04:43.333884 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.333919 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:43.334433 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:43.335821 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:43.335825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:43.336046 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:43.336212 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:43.336226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.337585 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.337591 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:43.338531 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:43.338541 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:43.339438 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:43.339442 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:43.339541 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:43.340443 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:43.340456 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:43.342221 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:43.344981 (+ 2760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:43.344984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:43.344986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:43.345515 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:43.346077 (+ 562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:43.346081 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:43.346084 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:43.346350 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:43.347242 (+ 892us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:43.347251 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:43.347724 (+ 473us) client_negotiation.cc:769] Sending connection context
0620 06:04:43.347988 (+ 264us) client_negotiation.cc:240] Negotiation successful
0620 06:04:43.348544 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":372,"spinlock_wait_cycles":41728}
I20260620 06:04:43.350239 18524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad0f812f3984ac489c446e22798efd7" candidate_uuid: "edd17449f93d41349906e355b3c3a5be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" is_pre_election: true
I20260620 06:04:43.350639 18524 raft_consensus.cc:2454] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edd17449f93d41349906e355b3c3a5be in term 1.
I20260620 06:04:43.351930 18593 leader_election.cc:302] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6120ff42da8c49c2bbf9aa7b6966183f, edd17449f93d41349906e355b3c3a5be; no voters:
W20260620 06:04:43.352154 18591 leader_election.cc:341] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801): Not found: Tablet not found: fad0f812f3984ac489c446e22798efd7
I20260620 06:04:43.352242 18846 raft_consensus.cc:2781] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 06:04:43.352363 18846 raft_consensus.cc:482] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:04:43.352429 18846 raft_consensus.cc:3037] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:04:43.354895 18846 raft_consensus.cc:504] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:43.355379 18846 leader_election.cc:288] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [CANDIDATE]: Term 2 election: Requested vote from peers 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801), 6120ff42da8c49c2bbf9aa7b6966183f (127.3.115.1:34795)
I20260620 06:04:43.356087 18524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad0f812f3984ac489c446e22798efd7" candidate_uuid: "edd17449f93d41349906e355b3c3a5be" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "6120ff42da8c49c2bbf9aa7b6966183f"
W20260620 06:04:43.356196 18591 leader_election.cc:341] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801): Not found: Tablet not found: fad0f812f3984ac489c446e22798efd7
I20260620 06:04:43.356262 18524 raft_consensus.cc:3037] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:04:43.358770 18524 raft_consensus.cc:2454] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd17449f93d41349906e355b3c3a5be in term 2.
I20260620 06:04:43.359323 18593 leader_election.cc:302] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6120ff42da8c49c2bbf9aa7b6966183f, edd17449f93d41349906e355b3c3a5be; no voters: 5c38c30657914004b51cc15231259f02
I20260620 06:04:43.359553 18846 raft_consensus.cc:2781] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:04:43.359762 18846 raft_consensus.cc:686] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [term 2 LEADER]: Becoming Leader. State: Replica: edd17449f93d41349906e355b3c3a5be, State: Running, Role: LEADER
I20260620 06:04:43.360164 18846 consensus_queue.cc:237] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:43.360611 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.360324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:39877)
0620 06:04:43.360461 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.360531 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260620 06:04:43.823365 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.822912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:60033)
0620 06:04:43.823186 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.823252 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":75,"threads_started":1}
W20260620 06:04:43.845338 18591 consensus_peers.cc:487] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be -> Peer 5c38c30657914004b51cc15231259f02 (127.3.115.3:44801): Couldn't send request to peer 5c38c30657914004b51cc15231259f02. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fad0f812f3984ac489c446e22798efd7. This is attempt 1: this message will repeat every 5th retry.
I20260620 06:04:43.980175 18524 raft_consensus.cc:1261] T fad0f812f3984ac489c446e22798efd7 P 6120ff42da8c49c2bbf9aa7b6966183f [term 2 FOLLOWER]: Refusing update from remote peer edd17449f93d41349906e355b3c3a5be: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260620 06:04:43.980995 18846 consensus_queue.cc:1035] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be [LEADER]: Connected to new peer: Peer: permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260620 06:04:43.983145 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.982870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:56109)
0620 06:04:43.983016 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.983079 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 06:04:43.985643 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.985293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:36427)
0620 06:04:43.985481 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.985548 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260620 06:04:43.985947 18852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:43.985670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:43375)
0620 06:04:43.985838 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:43.985890 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260620 06:04:44.197904 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.197367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:56947)
0620 06:04:44.197673 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.197751 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260620 06:04:44.317526 18860 ts_tablet_manager.cc:927] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Initiating tablet copy from peer edd17449f93d41349906e355b3c3a5be (127.3.115.2:33621)
I20260620 06:04:44.318207 18860 tablet_copy_client.cc:316] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: tablet copy: Beginning tablet copy session from remote peer at address 127.3.115.2:33621
Jun 20 06:04:44 dist-test-slave-4h14 krb5kdc[17260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935482, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.2@KRBTEST.COM
I20260620 06:04:44.335178 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.318841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:33621 (local address 127.3.115.3:48593)
0620 06:04:44.319058 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.319079 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:44.319198 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:44.319856 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:44.319861 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:44.320268 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:44.320387 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:44.320425 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:44.321662 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:44.321665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:44.322405 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:44.322415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:44.323122 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:44.323125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:44.323208 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:44.328995 (+ 5787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:44.329012 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:44.331061 (+ 2049us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:44.333449 (+ 2388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:44.333460 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:44.333466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:44.333721 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:44.334063 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:44.334065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:44.334066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:44.334143 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:44.334616 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:44.334621 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:44.334741 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 06:04:44.334866 (+ 125us) client_negotiation.cc:240] Negotiation successful
0620 06:04:44.335048 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 06:04:44.335253 18862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.319002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:48593 (local address 127.3.115.2:33621)
0620 06:04:44.319341 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:44.319351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:44.319373 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:04:44.319423 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:44.319429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:44.319618 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:44.319751 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:44.320578 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:44.321535 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:44.322532 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:44.322973 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:44.323092 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:44.331249 (+ 8157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:44.331262 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:44.331277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:44.331308 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:44.333311 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:44.333860 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:44.333867 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:44.333868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:44.333971 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:44.334224 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:44.334227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:44.334228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:44.334513 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:44.334626 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:44.334980 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 06:04:44.335124 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":85,"threads_started":1}
I20260620 06:04:44.336396 18679 tablet_copy_service.cc:140] P edd17449f93d41349906e355b3c3a5be: Received BeginTabletCopySession request for tablet fad0f812f3984ac489c446e22798efd7 from peer 5c38c30657914004b51cc15231259f02 ({username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:48593)
I20260620 06:04:44.336588 18679 tablet_copy_service.cc:161] P edd17449f93d41349906e355b3c3a5be: Beginning new tablet copy session on tablet fad0f812f3984ac489c446e22798efd7 from peer 5c38c30657914004b51cc15231259f02 at {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:48593: session id = 5c38c30657914004b51cc15231259f02-fad0f812f3984ac489c446e22798efd7
I20260620 06:04:44.337836 18679 tablet_copy_source_session.cc:205] T fad0f812f3984ac489c446e22798efd7 P edd17449f93d41349906e355b3c3a5be: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 06:04:44.342257 18860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad0f812f3984ac489c446e22798efd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:44.349606 18860 tablet_copy_client.cc:791] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: tablet copy: Starting download of 0 data blocks...
I20260620 06:04:44.349831 18860 tablet_copy_client.cc:655] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: tablet copy: Starting download of 1 WAL segments...
I20260620 06:04:44.351794 18860 tablet_copy_client.cc:531] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 06:04:44.353822 18860 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Bootstrap starting.
I20260620 06:04:44.359329 18860 log.cc:824] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:44.366147 18860 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:04:44.366429 18860 tablet_bootstrap.cc:492] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Bootstrap complete.
I20260620 06:04:44.366704 18860 ts_tablet_manager.cc:1397] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.003s
I20260620 06:04:44.368981 18860 raft_consensus.cc:348] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:44.369228 18860 raft_consensus.cc:729] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c38c30657914004b51cc15231259f02, State: Initialized, Role: FOLLOWER
I20260620 06:04:44.369608 18860 consensus_queue.cc:260] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd17449f93d41349906e355b3c3a5be" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 33621 } } peers { permanent_uuid: "5c38c30657914004b51cc15231259f02" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44801 } } peers { permanent_uuid: "6120ff42da8c49c2bbf9aa7b6966183f" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 34795 } }
I20260620 06:04:44.370507 18860 ts_tablet_manager.cc:1428] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260620 06:04:44.370821 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.370530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:48937)
0620 06:04:44.370671 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.370732 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:04:44.371323 18679 tablet_copy_service.cc:342] P edd17449f93d41349906e355b3c3a5be: Request end of tablet copy session 5c38c30657914004b51cc15231259f02-fad0f812f3984ac489c446e22798efd7 received from {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:48593
I20260620 06:04:44.371459 18679 tablet_copy_service.cc:433] P edd17449f93d41349906e355b3c3a5be: ending tablet copy session 5c38c30657914004b51cc15231259f02-fad0f812f3984ac489c446e22798efd7 on tablet fad0f812f3984ac489c446e22798efd7 with peer 5c38c30657914004b51cc15231259f02
I20260620 06:04:44.371670 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.371410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:36901)
0620 06:04:44.371558 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.371615 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260620 06:04:44.806522 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.806126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:37609)
0620 06:04:44.806378 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.806429 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260620 06:04:44.806695 18793 raft_consensus.cc:1203] T fad0f812f3984ac489c446e22798efd7 P 5c38c30657914004b51cc15231259f02 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260620 06:04:44.807703 18859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.807426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:60663)
0620 06:04:44.807551 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.807613 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260620 06:04:44.987174 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:44.986698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:55275)
0620 06:04:44.987008 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:44.987071 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":78,"threads_started":1}
I20260620 06:04:45.807721 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:45.807271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:41109)
0620 06:04:45.807547 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:45.807623 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":76,"threads_started":1}
I20260620 06:04:45.808902 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:45.808395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:52723)
0620 06:04:45.808705 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:45.808769 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":88,"threads_started":1}
I20260620 06:04:45.988539 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:45.988059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:38219)
0620 06:04:45.988371 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:45.988440 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":83,"threads_started":1}
I20260620 06:04:46.809059 18871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:46.808615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:44643)
0620 06:04:46.808894 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:46.808958 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260620 06:04:46.810014 18872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:46.809610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:60153)
0620 06:04:46.809853 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:46.809905 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
I20260620 06:04:46.990321 18873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:46.989621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:34025)
0620 06:04:46.990110 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:46.990190 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":182,"threads_started":1}
I20260620 06:04:47.810797 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:47.810176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:53023)
0620 06:04:47.810573 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:47.810674 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":112,"threads_started":1}
I20260620 06:04:47.811554 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:47.811155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:34813)
0620 06:04:47.811415 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:47.811470 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260620 06:04:47.811685 18874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:47.811418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:36737)
0620 06:04:47.811534 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:47.811592 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:04:47.991808 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:47.991270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:35389)
0620 06:04:47.991591 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:47.991649 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":93,"threads_started":1}
I20260620 06:04:48.812953 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:48.812424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:49411)
0620 06:04:48.812768 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:48.812849 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":82,"threads_started":1}
I20260620 06:04:48.813032 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:48.812510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:45015)
0620 06:04:48.812846 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:48.812909 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
I20260620 06:04:48.993255 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:48.992840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:49669)
0620 06:04:48.993098 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:48.993160 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":81,"threads_started":1}
I20260620 06:04:49.814313 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:49.813793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:36977)
0620 06:04:49.814151 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:49.814211 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":122,"threads_started":1}
I20260620 06:04:49.814323 18881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:49.813883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:35707)
0620 06:04:49.814179 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:49.814234 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":101,"threads_started":1}
I20260620 06:04:49.994589 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:49.994128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:37835)
0620 06:04:49.994408 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:49.994479 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260620 06:04:50.815614 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:50.815135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:36451)
0620 06:04:50.815438 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:50.815513 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":91,"threads_started":1}
I20260620 06:04:50.815644 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:50.815208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:38903)
0620 06:04:50.815501 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:50.815554 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":109,"threads_started":1}
I20260620 06:04:50.996083 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:50.995560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:39157)
0620 06:04:50.995907 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:50.995981 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":88,"threads_started":1}
I20260620 06:04:51.816952 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:51.816476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:38187)
0620 06:04:51.816797 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:51.816857 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":121,"threads_started":1}
I20260620 06:04:51.816995 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:51.816518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:58189)
0620 06:04:51.816811 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:51.816904 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":107,"threads_started":1}
I20260620 06:04:51.997532 18888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:51.997071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:34375)
0620 06:04:51.997355 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:51.997421 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":121,"threads_started":1}
I20260620 06:04:52.818279 18890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:52.817864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:43629)
0620 06:04:52.818135 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:52.818190 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":90,"threads_started":1}
I20260620 06:04:52.818279 18889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:52.817824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:58929)
0620 06:04:52.818127 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:52.818182 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":140,"threads_started":1}
I20260620 06:04:52.998860 18891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:52.998375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:56615)
0620 06:04:52.998690 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:52.998758 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":101,"threads_started":1}
I20260620 06:04:53.819603 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:53.819137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:50541)
0620 06:04:53.819425 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:53.819504 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":79,"threads_started":1}
I20260620 06:04:53.819633 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:53.819193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:60953)
0620 06:04:53.819489 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:53.819546 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":110,"threads_started":1}
I20260620 06:04:54.000267 18894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:53.999748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:51429)
0620 06:04:54.000080 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:54.000153 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":80,"threads_started":1}
I20260620 06:04:54.820803 18895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:54.820297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:40945)
0620 06:04:54.820583 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:54.820696 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":82,"threads_started":1}
I20260620 06:04:54.820842 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:54.820382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:58967)
0620 06:04:54.820649 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:54.820757 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":79,"threads_started":1}
I20260620 06:04:55.001556 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:55.001141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:52333)
0620 06:04:55.001391 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:55.001456 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":80,"threads_started":1}
I20260620 06:04:55.822252 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:55.821800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:43571)
0620 06:04:55.822095 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:55.822154 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":110,"threads_started":1}
I20260620 06:04:55.822284 18899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:55.821824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:33695)
0620 06:04:55.822122 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:55.822179 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
I20260620 06:04:56.002879 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:56.002394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:50343)
0620 06:04:56.002704 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:56.002776 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":96,"threads_started":1}
I20260620 06:04:56.823818 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:56.823344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.3:50117)
0620 06:04:56.823616 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:56.823690 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":100,"threads_started":1}
I20260620 06:04:56.823972 18902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:56.823362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.2:58695)
0620 06:04:56.823772 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:56.823824 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":87,"threads_started":1}
I20260620 06:04:57.004290 18903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:57.003870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37891 (local address 127.3.115.1:33897)
0620 06:04:57.004133 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:57.004192 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:37891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":78,"threads_started":1}
I20260620 06:04:57.201787 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 18406
I20260620 06:04:57.207904 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 18573
I20260620 06:04:57.213804 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 18708
2026-06-20T06:04:57Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19546 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:04:57 dist-test-slave-4h14 krb5kdc[18906](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:04:57 dist-test-slave-4h14 krb5kdc[18906](info): set up 2 sockets
Jun 20 06:04:57 dist-test-slave-4h14 krb5kdc[18906](info): commencing operation
krb5kdc: starting...
W20260620 06:04:58.265007 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:04:58 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:04:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:04:58.389511 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42915
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:58.506855 18919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:58.507207 18919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:58.507292 18919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:58.511502 18919 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:04:58.511605 18919 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:04:58.511633 18919 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:04:58.511653 18919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:04:58.511698 18919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:58.511762 18919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:58.511802 18919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:04:58.511835 18919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:04:58.517516 18919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:58.518703 18919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:58.519780 18919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:58.529186 18927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:58.529187 18924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:58.529254 18919 server_base.cc:1034] running on GCE node
W20260620 06:04:58.529187 18925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:58.530109 18919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:58.531261 18919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:58.532441 18919 hybrid_clock.cc:648] HybridClock initialized: now 1781935498532422 us; error 34 us; skew 500 ppm
Jun 20 06:04:58 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:58.535593 18919 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:04:58.536590 18919 webserver.cc:460] Webserver started at http://127.3.115.62:38463/ using document root <none> and password file <none>
I20260620 06:04:58.537243 18919 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:58.537307 18919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:58.537533 18919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:58.539674 18919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "fdd14aac1ee04cbe90d86845fb5ae7de"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "dff16a5d7c2db19304153699f54ce39b"
server_key_iv: "acf5edd665aab3281e280f6d4129d119"
server_key_version: "encryptionkey@0"
I20260620 06:04:58.540256 18919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "fdd14aac1ee04cbe90d86845fb5ae7de"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "dff16a5d7c2db19304153699f54ce39b"
server_key_iv: "acf5edd665aab3281e280f6d4129d119"
server_key_version: "encryptionkey@0"
I20260620 06:04:58.544215 18919 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 06:04:58.547044 18934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:58.548470 18919 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:04:58.548638 18919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "fdd14aac1ee04cbe90d86845fb5ae7de"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "dff16a5d7c2db19304153699f54ce39b"
server_key_iv: "acf5edd665aab3281e280f6d4129d119"
server_key_version: "encryptionkey@0"
I20260620 06:04:58.548821 18919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:58.566495 18919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:58.570273 18919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:58.570500 18919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:58.578176 18919 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42915
I20260620 06:04:58.578562 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42915 every 8 connection(s)
I20260620 06:04:58.579242 18919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:04:58.582183 18987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:58.586373 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 18919
I20260620 06:04:58.586488 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:04:58.586826 3532 external_mini_cluster.cc:1428] Setting key f5db407756079bb92e3f1cb3df66c9b1
I20260620 06:04:58.588449 18987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap starting.
I20260620 06:04:58.591104 18987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:58.591886 18987 log.cc:824] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:58.594225 18987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: No bootstrap required, opened a new log
Jun 20 06:04:58 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:58.598830 18987 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:04:58.599116 18987 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:58.599203 18987 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd14aac1ee04cbe90d86845fb5ae7de, State: Initialized, Role: FOLLOWER
I20260620 06:04:58.599740 18987 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:04:58.599915 18987 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:04:58.599989 18987 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:04:58.600092 18987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:58.601152 18987 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:04:58.601596 18987 leader_election.cc:302] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd14aac1ee04cbe90d86845fb5ae7de; no voters:
I20260620 06:04:58.601966 18987 leader_election.cc:288] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:04:58.602087 18992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:58.602305 18992 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 LEADER]: Becoming Leader. State: Replica: fdd14aac1ee04cbe90d86845fb5ae7de, State: Running, Role: LEADER
I20260620 06:04:58.602700 18992 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:04:58.602838 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:58.588348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54452 (local address 127.3.115.62:42915)
0620 06:04:58.588795 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:58.588813 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:58.588851 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:04:58.589528 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:58.589546 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:58.589893 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:58.590199 (+ 306us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:58.591153 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:58.592287 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:58.593504 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:58.594041 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:58.594169 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:58.597110 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:58.597116 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:58.597132 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:58.597163 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:58.599097 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:58.599655 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:58.599660 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:58.599667 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:58.599793 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:58.600260 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:58.600263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:58.600264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:58.600779 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:58.600998 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:58.601302 (+ 304us) server_negotiation.cc:299] Negotiation successful
0620 06:04:58.601635 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":140,"threads_started":1}
I20260620 06:04:58.603451 18987 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:04:58.604410 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } } }
I20260620 06:04:58.604533 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: This master's current role is: LEADER
I20260620 06:04:58.604777 18993 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd14aac1ee04cbe90d86845fb5ae7de. Latest consensus state: current_term: 1 leader_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } } }
I20260620 06:04:58.604846 18993 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: This master's current role is: LEADER
W20260620 06:04:58.608615 19007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:04:58.608718 19007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:04:58.608774 19001 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:04:58.609499 19001 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:04:58.615187 19001 catalog_manager.cc:1347] Generated new cluster ID: b793f2929f8342aea69716102d28fd8a
I20260620 06:04:58.615269 19001 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:04:58.624217 19001 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:04:58.625579 19001 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:04:58.631908 19001 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Generated new TSK 0
I20260620 06:04:58.632901 19001 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:04:58.668483 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42915
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:58.785218 19013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:58.785514 19013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:58.785595 19013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:58.789659 19013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:58.789757 19013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:58.789851 19013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:04:58.795166 19013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:58.796289 19013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:58.797389 19013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:58.807261 19018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:58.807279 19019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:58.807261 19021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:58.807510 19013 server_base.cc:1034] running on GCE node
I20260620 06:04:58.808022 19013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:58.808773 19013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:58.809937 19013 hybrid_clock.cc:648] HybridClock initialized: now 1781935498809921 us; error 31 us; skew 500 ppm
Jun 20 06:04:58 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:58.812882 19013 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:04:58.813759 19013 webserver.cc:460] Webserver started at http://127.3.115.1:33755/ using document root <none> and password file <none>
I20260620 06:04:58.814344 19013 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:58.814440 19013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:58.814683 19013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:58.816936 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "81559fa0905cd4a11b06ff9debec9bc9"
server_key_iv: "152ccd56c1fc9c7ee2df21b5adfec528"
server_key_version: "encryptionkey@0"
I20260620 06:04:58.817504 19013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "81559fa0905cd4a11b06ff9debec9bc9"
server_key_iv: "152ccd56c1fc9c7ee2df21b5adfec528"
server_key_version: "encryptionkey@0"
I20260620 06:04:58.821313 19013 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:04:58.823925 19028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:58.825395 19013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:04:58.825542 19013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "81559fa0905cd4a11b06ff9debec9bc9"
server_key_iv: "152ccd56c1fc9c7ee2df21b5adfec528"
server_key_version: "encryptionkey@0"
I20260620 06:04:58.825670 19013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:58.843108 19013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:58.846418 19013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:58.846638 19013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:58.847272 19013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:58.848130 19013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:58.848201 19013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:58.848268 19013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:58.848309 19013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:58.857995 19013 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35603
I20260620 06:04:58.858012 19141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35603 every 8 connection(s)
I20260620 06:04:58.858934 19013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:04:58.866022 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19013
I20260620 06:04:58.866135 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:04:58.866531 3532 external_mini_cluster.cc:1428] Setting key ab7fb58aba76fe8b312cd5b7c1c6b1e3
Jun 20 06:04:58 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935498, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:58.874137 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:58.861120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:51165 (local address 127.3.115.62:42915)
0620 06:04:58.861273 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:58.861278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:58.862214 (+ 936us) server_negotiation.cc:407] Connection header received
0620 06:04:58.863517 (+ 1303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:58.863520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:58.863571 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:58.863660 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:58.865029 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:58.865573 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:58.867015 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:58.867250 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:58.867375 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:58.870351 (+ 2976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:58.870354 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:58.870356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:58.870384 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:58.871924 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:58.872502 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:58.872506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:58.872507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:58.872588 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:58.872971 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:58.872977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:58.872982 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:58.873180 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:58.873289 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:58.873832 (+ 543us) server_negotiation.cc:299] Negotiation successful
0620 06:04:58.873966 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:04:58.875058 19144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:58.861544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:51165)
0620 06:04:58.862044 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:58.862091 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:58.863281 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:58.863832 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:58.863839 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:58.864285 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:58.864825 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:58.864840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:58.865720 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:58.865723 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:58.866853 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:58.866864 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:58.867442 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:58.867447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:58.867662 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:58.868287 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:58.868304 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:58.870157 (+ 1853us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:58.872058 (+ 1901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:58.872065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:58.872083 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:58.872372 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:58.872703 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:58.872707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:58.872710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:58.872872 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:58.873288 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:58.873295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:58.873573 (+ 278us) client_negotiation.cc:769] Sending connection context
0620 06:04:58.873776 (+ 203us) client_negotiation.cc:240] Negotiation successful
0620 06:04:58.874155 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:04:58.876492 19142 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42915
I20260620 06:04:58.876816 19142 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:58.877357 19142 heartbeater.cc:511] Master 127.3.115.62:42915 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:04:58.879000 18951 ts_manager.cc:194] Registered new tserver with Master: d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603)
I20260620 06:04:58.880189 18951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:51165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:04:58.895879 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42915
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:59.010851 19147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:59.011161 19147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:59.011207 19147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:59.015560 19147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:59.015635 19147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:59.015733 19147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:04:59.021211 19147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:59.022439 19147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:59.023540 19147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:59.033541 19152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:59.033567 19153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:59.033567 19155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:59.033943 19147 server_base.cc:1034] running on GCE node
I20260620 06:04:59.034351 19147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:59.034969 19147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:59.036144 19147 hybrid_clock.cc:648] HybridClock initialized: now 1781935499036104 us; error 54 us; skew 500 ppm
Jun 20 06:04:59 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:59.044011 19147 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:04:59.045094 19147 webserver.cc:460] Webserver started at http://127.3.115.2:45043/ using document root <none> and password file <none>
I20260620 06:04:59.045701 19147 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:59.045766 19147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:59.046007 19147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:59.048177 19147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "4b039a93c2844b5babea8061b4eda7e3"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "75be4115556fb285d5d15b138ce911d7"
server_key_iv: "50fd3c6171d0c92f8eca57bf0a1e6c57"
server_key_version: "encryptionkey@0"
I20260620 06:04:59.048831 19147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "4b039a93c2844b5babea8061b4eda7e3"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "75be4115556fb285d5d15b138ce911d7"
server_key_iv: "50fd3c6171d0c92f8eca57bf0a1e6c57"
server_key_version: "encryptionkey@0"
I20260620 06:04:59.052721 19147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:04:59.055192 19162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:59.056641 19147 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:04:59.056825 19147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "4b039a93c2844b5babea8061b4eda7e3"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "75be4115556fb285d5d15b138ce911d7"
server_key_iv: "50fd3c6171d0c92f8eca57bf0a1e6c57"
server_key_version: "encryptionkey@0"
I20260620 06:04:59.056926 19147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:59.074103 19147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:59.077188 19147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:59.077386 19147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:59.078022 19147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:59.078863 19147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:59.078910 19147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:59.078976 19147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:59.079010 19147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:59.088194 19147 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:37415
I20260620 06:04:59.088217 19275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:37415 every 8 connection(s)
I20260620 06:04:59.089210 19147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:04:59.092856 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19147
I20260620 06:04:59.092981 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:04:59.093290 3532 external_mini_cluster.cc:1428] Setting key 5f946b3f7f4598affffb7139a6c33bfd
Jun 20 06:04:59 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:04:59.103812 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.091089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:46571 (local address 127.3.115.62:42915)
0620 06:04:59.091219 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.091228 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.092085 (+ 857us) server_negotiation.cc:407] Connection header received
0620 06:04:59.093632 (+ 1547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.093636 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.093698 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.093805 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:59.094984 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.095534 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.096683 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.096910 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.097027 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.099729 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:59.099737 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:59.099742 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:59.099777 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:59.101546 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:59.102190 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:59.102194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:59.102198 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:59.102289 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:59.102614 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:59.102618 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:59.102622 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:59.102884 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:59.103011 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.103545 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.103685 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:04:59.104775 19278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.091416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:46571)
0620 06:04:59.091905 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.091961 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.093396 (+ 1435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.093988 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.093995 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.094348 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:59.094839 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.094852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.095657 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.095661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.096534 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.096543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.097063 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.097067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.097242 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.097821 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:59.097847 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:59.099550 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:59.101686 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:59.101693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:59.101711 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:59.102085 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:59.102382 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:59.102384 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:59.102385 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:59.102513 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:59.103014 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:59.103038 (+ 24us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:59.103269 (+ 231us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.103484 (+ 215us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.103744 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:04:59.105989 19276 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42915
I20260620 06:04:59.106262 19276 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:59.106783 19276 heartbeater.cc:511] Master 127.3.115.62:42915 requested a full tablet report, sending...
I20260620 06:04:59.107834 18951 ts_manager.cc:194] Registered new tserver with Master: 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415)
I20260620 06:04:59.108410 18951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:46571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:04:59.123515 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42915
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:04:59.253616 19281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:04:59.253922 19281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:04:59.253969 19281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:04:59.258162 19281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:04:59.258232 19281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:04:59.258342 19281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:04:59.264007 19281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:04:59.265141 19281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:04:59.266203 19281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:04:59.275942 19289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:04:59.275873 19287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:59.276011 19281 server_base.cc:1034] running on GCE node
W20260620 06:04:59.275873 19286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:04:59.276569 19281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:04:59.277240 19281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:04:59.278398 19281 hybrid_clock.cc:648] HybridClock initialized: now 1781935499278384 us; error 27 us; skew 500 ppm
Jun 20 06:04:59 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:04:59.281833 19281 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:04:59.283207 19281 webserver.cc:460] Webserver started at http://127.3.115.3:37909/ using document root <none> and password file <none>
I20260620 06:04:59.284042 19281 fs_manager.cc:359] Metadata directory not provided
I20260620 06:04:59.284153 19281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:04:59.284437 19281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:04:59.287392 19281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "b5f6ee5332ef453db565c39700cb40f7"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "5b6d04775ee6efa54f0d6eaa59dc7495"
server_key_iv: "d1b2cbb02b47c99dbb921373f44d750f"
server_key_version: "encryptionkey@0"
I20260620 06:04:59.288113 19281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "b5f6ee5332ef453db565c39700cb40f7"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "5b6d04775ee6efa54f0d6eaa59dc7495"
server_key_iv: "d1b2cbb02b47c99dbb921373f44d750f"
server_key_version: "encryptionkey@0"
I20260620 06:04:59.292244 19281 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:04:59.294915 19296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:59.296252 19281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:04:59.296388 19281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "b5f6ee5332ef453db565c39700cb40f7"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "5b6d04775ee6efa54f0d6eaa59dc7495"
server_key_iv: "d1b2cbb02b47c99dbb921373f44d750f"
server_key_version: "encryptionkey@0"
I20260620 06:04:59.296507 19281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:04:59.324581 19281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:04:59.328148 19281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:04:59.328356 19281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:04:59.329043 19281 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:04:59.329994 19281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:04:59.330096 19281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:59.330189 19281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:04:59.330226 19281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:04:59.339267 19281 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40497
I20260620 06:04:59.339287 19409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40497 every 8 connection(s)
I20260620 06:04:59.340314 19281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
Jun 20 06:04:59 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935499, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:04:59.350479 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19281
I20260620 06:04:59.350631 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:04:59.351014 3532 external_mini_cluster.cc:1428] Setting key 71472e5d74ccc58f6527448073f65ebf
I20260620 06:04:59.354411 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.342242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:60141 (local address 127.3.115.62:42915)
0620 06:04:59.342394 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.342399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.343750 (+ 1351us) server_negotiation.cc:407] Connection header received
0620 06:04:59.344592 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.344596 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.344645 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.344777 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:59.345983 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.346508 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.347640 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.347864 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.347946 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.350456 (+ 2510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:59.350459 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:59.350461 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:59.350486 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:59.352190 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:59.352852 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:59.352855 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:59.352857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:59.352948 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:59.353392 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:59.353396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:59.353398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:59.353607 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:59.353686 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.354194 (+ 508us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.354297 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:04:59.355309 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.342705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:60141)
0620 06:04:59.343590 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.343636 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.344402 (+ 766us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.344950 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.344957 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.345333 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:04:59.345841 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.345855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.346617 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.346620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.347506 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.347515 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.348009 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.348012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.348165 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.348796 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:04:59.348812 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:04:59.350262 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:04:59.352326 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:59.352330 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:59.352344 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:59.352754 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:59.353043 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:04:59.353045 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:04:59.353047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:04:59.353286 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:04:59.353707 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:04:59.353712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:04:59.353957 (+ 245us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.354147 (+ 190us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.354419 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"thread_start_us":532,"threads_started":1}
I20260620 06:04:59.356331 19410 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42915
I20260620 06:04:59.356539 19410 heartbeater.cc:464] Registering TS with master...
I20260620 06:04:59.357064 19410 heartbeater.cc:511] Master 127.3.115.62:42915 requested a full tablet report, sending...
I20260620 06:04:59.358004 18951 ts_manager.cc:194] Registered new tserver with Master: b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
I20260620 06:04:59.358583 18951 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:60141
I20260620 06:04:59.365144 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:04:59.374366 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.367104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.3.115.62:42915)
0620 06:04:59.367271 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.367276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.367435 (+ 159us) server_negotiation.cc:407] Connection header received
0620 06:04:59.367529 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.367531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.367573 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.367640 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:04:59.368273 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.368890 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.369711 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.369934 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.370024 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.371403 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:04:59.371405 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:04:59.371407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:04:59.371430 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:04:59.372871 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:59.373267 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:59.373270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:59.373271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:59.373346 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:04:59.373570 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:04:59.373572 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:04:59.373573 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:04:59.373778 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:04:59.373877 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.374076 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.374169 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 06:04:59.377439 18951 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:04:59.380420 18951 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:04:59.394506 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.388472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:37415 (local address 127.0.0.1:38240)
0620 06:04:59.388892 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.388936 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.389241 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.390324 (+ 1083us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.390330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.390368 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.390620 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.390629 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.391656 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.391661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.393000 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.393010 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.393984 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.393988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.394090 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.394126 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.394236 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.394367 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":104,"spinlock_wait_cycles":19200,"thread_start_us":101,"threads_started":1}
I20260620 06:04:59.394515 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.388576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38240 (local address 127.3.115.2:37415)
0620 06:04:59.389814 (+ 1238us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.389826 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.389847 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:04:59.389920 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.389925 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.390162 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.390339 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.390760 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.391516 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.393128 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.393862 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.393985 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.394150 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.394260 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.394354 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1141,"thread_start_us":138,"threads_started":1}
I20260620 06:04:59.395040 19422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.388565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40497 (local address 127.0.0.1:56850)
0620 06:04:59.389485 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.389498 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.389578 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.390323 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.390332 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.390371 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.390634 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.390643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.392072 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.392074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.393439 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.393449 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.394651 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.394654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.394732 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.394751 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.394812 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.394871 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":271,"spinlock_wait_cycles":2688,"thread_start_us":115,"threads_started":1}
I20260620 06:04:59.395229 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.389221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35603 (local address 127.0.0.1:33728)
0620 06:04:59.389674 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.389688 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.389771 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.390405 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.390409 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.390426 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.390664 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.390674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.392035 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.392038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.393296 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.393304 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.394379 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.394386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.394485 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.394506 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.394553 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.395109 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":88,"thread_start_us":93,"threads_started":1}
I20260620 06:04:59.395467 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.389209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.3.115.3:40497)
0620 06:04:59.389633 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.389645 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.389668 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:04:59.389753 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.389758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.389973 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.390140 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.390771 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.391918 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.393576 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.394347 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.394440 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.394592 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.395211 (+ 619us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.395314 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":96,"threads_started":1}
I20260620 06:04:59.395550 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.389636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33728 (local address 127.3.115.1:35603)
0620 06:04:59.389995 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.390001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.390023 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:04:59.390098 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.390102 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.390268 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.390601 (+ 333us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.390748 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.391911 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.393420 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.394233 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.394927 (+ 694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.395058 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.395105 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.395395 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":98,"threads_started":1}
I20260620 06:04:59.397958 19344 tablet_service.cc:1467] Processing CreateTablet for tablet d9df53a433014dfaa9490865581d60c7 (DEFAULT_TABLE table=test-table [id=e8764352fdd14f709396d31f5a1a4106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:59.398056 19076 tablet_service.cc:1467] Processing CreateTablet for tablet d9df53a433014dfaa9490865581d60c7 (DEFAULT_TABLE table=test-table [id=e8764352fdd14f709396d31f5a1a4106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:59.397926 19210 tablet_service.cc:1467] Processing CreateTablet for tablet d9df53a433014dfaa9490865581d60c7 (DEFAULT_TABLE table=test-table [id=e8764352fdd14f709396d31f5a1a4106]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:04:59.398924 19344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df53a433014dfaa9490865581d60c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:59.398941 19076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df53a433014dfaa9490865581d60c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:59.398941 19210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9df53a433014dfaa9490865581d60c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:04:59.406659 19427 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Bootstrap starting.
I20260620 06:04:59.406664 19428 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap starting.
I20260620 06:04:59.408644 19428 tablet_bootstrap.cc:654] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:59.408636 19427 tablet_bootstrap.cc:654] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:59.409155 19429 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Bootstrap starting.
I20260620 06:04:59.409658 19428 log.cc:824] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:59.409868 19427 log.cc:824] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:59.411953 19428 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: No bootstrap required, opened a new log
I20260620 06:04:59.411979 19429 tablet_bootstrap.cc:654] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Neither blocks nor log segments found. Creating new log.
I20260620 06:04:59.412036 19427 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: No bootstrap required, opened a new log
I20260620 06:04:59.412187 19428 ts_tablet_manager.cc:1397] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 06:04:59.412230 19427 ts_tablet_manager.cc:1397] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260620 06:04:59.412895 19429 log.cc:824] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:04:59.415302 19429 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: No bootstrap required, opened a new log
I20260620 06:04:59.415526 19429 ts_tablet_manager.cc:1397] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:04:59.416123 19428 raft_consensus.cc:348] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.416376 19428 raft_consensus.cc:374] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:59.416428 19428 raft_consensus.cc:729] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49a72e9b33044ed80bd0f6efba7d6ed, State: Initialized, Role: FOLLOWER
I20260620 06:04:59.416381 19427 raft_consensus.cc:348] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.416692 19427 raft_consensus.cc:374] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:59.416750 19427 raft_consensus.cc:729] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f6ee5332ef453db565c39700cb40f7, State: Initialized, Role: FOLLOWER
I20260620 06:04:59.416916 19428 consensus_queue.cc:260] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.417168 19427 consensus_queue.cc:260] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.417689 19142 heartbeater.cc:503] Master 127.3.115.62:42915 was elected leader, sending a full tablet report...
I20260620 06:04:59.418002 19428 ts_tablet_manager.cc:1428] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:04:59.418103 19427 ts_tablet_manager.cc:1428] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:04:59.418426 19410 heartbeater.cc:503] Master 127.3.115.62:42915 was elected leader, sending a full tablet report...
I20260620 06:04:59.419421 19429 raft_consensus.cc:348] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.419721 19429 raft_consensus.cc:374] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:04:59.419785 19429 raft_consensus.cc:729] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b039a93c2844b5babea8061b4eda7e3, State: Initialized, Role: FOLLOWER
I20260620 06:04:59.420199 19429 consensus_queue.cc:260] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.420858 19276 heartbeater.cc:503] Master 127.3.115.62:42915 was elected leader, sending a full tablet report...
I20260620 06:04:59.421213 19429 ts_tablet_manager.cc:1428] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:04:59.533836 19434 raft_consensus.cc:482] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:59.534080 19434 raft_consensus.cc:504] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.535354 19434 leader_election.cc:288] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603)
I20260620 06:04:59.539165 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.535588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:37415 (local address 127.3.115.3:37479)
0620 06:04:59.535783 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.535800 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.535898 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.536211 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.536214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.536231 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.536370 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.536375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.537176 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.537182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.538019 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.538029 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.538728 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.538731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.538841 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.538858 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.538947 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.539021 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"spinlock_wait_cycles":5376}
I20260620 06:04:59.539165 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.535731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:37479 (local address 127.3.115.2:37415)
0620 06:04:59.535886 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.535891 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.535910 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:04:59.536048 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.536051 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.536093 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.536199 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.536490 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.537066 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.538146 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.538605 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.538715 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.538778 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.538963 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.539040 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:04:59.539466 19426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.535820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:44191 (local address 127.3.115.1:35603)
0620 06:04:59.535946 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.535952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.536219 (+ 267us) server_negotiation.cc:407] Connection header received
0620 06:04:59.536405 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.536408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.536459 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.536549 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.536867 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.537349 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.538397 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.538823 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.538946 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.539007 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.539232 (+ 225us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.539312 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:04:59.539855 19436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.535713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35603 (local address 127.3.115.3:44191)
0620 06:04:59.536116 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.536128 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.536287 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.536559 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.536561 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.536576 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.536749 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.536756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.537482 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.537485 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.538269 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.538277 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.538954 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.538958 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.539075 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.539093 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.539595 (+ 502us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.539704 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":1,"spinlock_wait_cycles":16000,"thread_start_us":129,"threads_started":1}
I20260620 06:04:59.539898 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df53a433014dfaa9490865581d60c7" candidate_uuid: "b5f6ee5332ef453db565c39700cb40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b039a93c2844b5babea8061b4eda7e3" is_pre_election: true
I20260620 06:04:59.540225 19433 raft_consensus.cc:482] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:04:59.540333 19230 raft_consensus.cc:2454] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5f6ee5332ef453db565c39700cb40f7 in term 0.
I20260620 06:04:59.540412 19433 raft_consensus.cc:504] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.540575 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df53a433014dfaa9490865581d60c7" candidate_uuid: "b5f6ee5332ef453db565c39700cb40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" is_pre_election: true
I20260620 06:04:59.540872 19096 raft_consensus.cc:2454] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5f6ee5332ef453db565c39700cb40f7 in term 0.
I20260620 06:04:59.540989 19300 leader_election.cc:302] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7; no voters:
I20260620 06:04:59.541299 19434 raft_consensus.cc:2781] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:04:59.541381 19434 raft_consensus.cc:482] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:04:59.541428 19434 raft_consensus.cc:3037] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:59.541775 19433 leader_election.cc:288] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
I20260620 06:04:59.543345 19434 raft_consensus.cc:504] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.544034 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df53a433014dfaa9490865581d60c7" candidate_uuid: "b5f6ee5332ef453db565c39700cb40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b039a93c2844b5babea8061b4eda7e3"
I20260620 06:04:59.544098 19096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df53a433014dfaa9490865581d60c7" candidate_uuid: "b5f6ee5332ef453db565c39700cb40f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
I20260620 06:04:59.544184 19230 raft_consensus.cc:3037] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:59.544231 19096 raft_consensus.cc:3037] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:04:59.545502 19230 raft_consensus.cc:2454] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5f6ee5332ef453db565c39700cb40f7 in term 1.
I20260620 06:04:59.545609 19096 raft_consensus.cc:2454] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5f6ee5332ef453db565c39700cb40f7 in term 1.
I20260620 06:04:59.545835 19300 leader_election.cc:302] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7; no voters:
I20260620 06:04:59.546257 19439 raft_consensus.cc:2781] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:04:59.546373 19434 leader_election.cc:288] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [CANDIDATE]: Term 1 election: Requested vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603)
I20260620 06:04:59.546555 19423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.542069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:39633 (local address 127.3.115.2:37415)
0620 06:04:59.542179 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.542183 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.542406 (+ 223us) server_negotiation.cc:407] Connection header received
0620 06:04:59.542529 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.542534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.542580 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.542660 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.542917 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.543556 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.545208 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.545803 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.545968 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.546034 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.546335 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.546419 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:04:59.546572 19439 raft_consensus.cc:686] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 1 LEADER]: Becoming Leader. State: Replica: b5f6ee5332ef453db565c39700cb40f7, State: Running, Role: LEADER
I20260620 06:04:59.546937 19438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.541903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:37415 (local address 127.3.115.1:39633)
0620 06:04:59.542315 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.542327 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.542446 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.542680 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.542683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.542699 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.542815 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.542819 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.543681 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.543684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.545064 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.545074 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.546092 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.546095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.546194 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.546218 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.546677 (+ 459us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.546766 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":3,"thread_start_us":145,"threads_started":1}
I20260620 06:04:59.547199 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.542059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:50711 (local address 127.3.115.3:40497)
0620 06:04:59.542185 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.542189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.542528 (+ 339us) server_negotiation.cc:407] Connection header received
0620 06:04:59.542737 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.542740 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.542794 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.542839 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:04:59.543226 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.543884 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.545676 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.546348 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.546436 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.546487 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.547000 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.547079 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:04:59.546937 19437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.541991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40497 (local address 127.3.115.1:50711)
0620 06:04:59.542445 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.542461 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:04:59.542612 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:04:59.542977 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:04:59.542983 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:04:59.543003 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:04:59.543106 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.543112 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.544001 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.544005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.545477 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:04:59.545491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.546543 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.546547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:04:59.546649 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.546677 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:04:59.546746 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 06:04:59.546791 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":383,"mutex_wait_us":24,"thread_start_us":98,"threads_started":1}
I20260620 06:04:59.547338 19230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df53a433014dfaa9490865581d60c7" candidate_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b039a93c2844b5babea8061b4eda7e3" is_pre_election: true
I20260620 06:04:59.547502 19230 raft_consensus.cc:2379] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d49a72e9b33044ed80bd0f6efba7d6ed in current term 1: Already voted for candidate b5f6ee5332ef453db565c39700cb40f7 in this term.
I20260620 06:04:59.547473 19439 consensus_queue.cc:237] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:04:59.547878 19364 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9df53a433014dfaa9490865581d60c7" candidate_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f6ee5332ef453db565c39700cb40f7" is_pre_election: true
I20260620 06:04:59.548444 19030 leader_election.cc:302] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d49a72e9b33044ed80bd0f6efba7d6ed; no voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7
I20260620 06:04:59.548739 19433 raft_consensus.cc:2726] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260620 06:04:59.551093 18951 catalog_manager.cc:5495] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "b5f6ee5332ef453db565c39700cb40f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20260620 06:04:59.587378 19425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.583356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56860 (local address 127.3.115.3:40497)
0620 06:04:59.583493 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:04:59.583498 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:04:59.583515 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:04:59.583677 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:04:59.583681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:04:59.583747 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:04:59.583861 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:04:59.584141 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.584768 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.585760 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:04:59.586032 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:04:59.586170 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:04:59.586513 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:04:59.586944 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:04:59.587093 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0620 06:04:59.587161 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 06:04:59.587220 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 06:04:59.590342 19277 tablet.cc:2369] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:04:59.591234 19411 tablet.cc:2369] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:04:59.593688 19230 raft_consensus.cc:1261] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Refusing update from remote peer b5f6ee5332ef453db565c39700cb40f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:59.593775 19096 raft_consensus.cc:1261] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Refusing update from remote peer b5f6ee5332ef453db565c39700cb40f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:04:59.594348 19440 consensus_queue.cc:1035] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:59.594524 19439 consensus_queue.cc:1035] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:04:59.601373 19443 mvcc.cc:205] Tried to move back new op lower bound from 7298807806328586240 to 7298807806149758976. Current Snapshot: MvccSnapshot[applied={T|T < 7298807806328586240}]
I20260620 06:04:59.601598 19444 mvcc.cc:205] Tried to move back new op lower bound from 7298807806328586240 to 7298807806149758976. Current Snapshot: MvccSnapshot[applied={T|T < 7298807806328586240}]
I20260620 06:04:59.613281 18951 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:59.613459 18951 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:04:59.616145 18951 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Sending DeleteTablet for 3 replicas of tablet d9df53a433014dfaa9490865581d60c7
I20260620 06:04:59.617043 19344 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df53a433014dfaa9490865581d60c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:59 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:56850
I20260620 06:04:59.617153 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19013
I20260620 06:04:59.617118 19210 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df53a433014dfaa9490865581d60c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:59 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:38240
I20260620 06:04:59.618119 19452 tablet_replica.cc:330] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: stopping tablet replica
I20260620 06:04:59.618427 19452 raft_consensus.cc:2229] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:04:59.618806 19452 raft_consensus.cc:2258] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:59.619246 19453 tablet_replica.cc:330] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: stopping tablet replica
I20260620 06:04:59.619544 19453 raft_consensus.cc:2229] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:04:59.619932 19453 raft_consensus.cc:2258] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:04:59.621573 19452 ts_tablet_manager.cc:1905] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:04:59.621855 19453 ts_tablet_manager.cc:1905] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:04:59.624744 19453 ts_tablet_manager.cc:1918] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:59.624734 19452 ts_tablet_manager.cc:1918] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:04:59.624850 19453 log.cc:1196] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/d9df53a433014dfaa9490865581d60c7
I20260620 06:04:59.625000 19452 log.cc:1196] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/d9df53a433014dfaa9490865581d60c7
W20260620 06:04:59.625037 18936 connection.cc:535] client connection to 127.3.115.1:35603 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 06:04:59.625190 18936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:04:59.625344 19453 ts_tablet_manager.cc:1939] T d9df53a433014dfaa9490865581d60c7 P b5f6ee5332ef453db565c39700cb40f7: Deleting consensus metadata
I20260620 06:04:59.625613 19452 ts_tablet_manager.cc:1939] T d9df53a433014dfaa9490865581d60c7 P 4b039a93c2844b5babea8061b4eda7e3: Deleting consensus metadata
I20260620 06:04:59.625980 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19147
I20260620 06:04:59.628141 18936 catalog_manager.cc:4841] TS b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497): tablet d9df53a433014dfaa9490865581d60c7 (table test-table [id=e8764352fdd14f709396d31f5a1a4106]) successfully deleted
I20260620 06:04:59.632725 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19281
I20260620 06:04:59.634076 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.632954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35603 (local address 127.0.0.1:33740)
0620 06:04:59.633866 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.633951 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":817}
W20260620 06:04:59.634320 18936 catalog_manager.cc:4584] TS d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d9df53a433014dfaa9490865581d60c7: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35603: connect: Connection refused (error 111)
I20260620 06:04:59.634963 19424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:04:59.634661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:37415 (local address 127.0.0.1:38246)
0620 06:04:59.634814 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:04:59.634875 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.2:37415: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:04:59.639011 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 18919
I20260620 06:05:19.645967 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42915
--webserver_interface=127.3.115.62
--webserver_port=38463
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:05:19.761786 19455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:19.762064 19455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:19.762107 19455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:19.766291 19455 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:05:19.766358 19455 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:05:19.766376 19455 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:05:19.766389 19455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:05:19.766408 19455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:19.766427 19455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:19.766443 19455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:05:19.766460 19455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:05:19.771865 19455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:19.772961 19455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:19.773952 19455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:19.787279 19464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:19.787281 19461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:19.787279 19462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:19.787640 19455 server_base.cc:1034] running on GCE node
I20260620 06:05:19.788231 19455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:19.789510 19455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:19.790695 19455 hybrid_clock.cc:648] HybridClock initialized: now 1781935519790678 us; error 30 us; skew 500 ppm
Jun 20 06:05:19 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:19.793821 19455 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:05:19.794799 19455 webserver.cc:460] Webserver started at http://127.3.115.62:38463/ using document root <none> and password file <none>
I20260620 06:05:19.795414 19455 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:19.795517 19455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:19.799612 19455 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:05:19.801925 19471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:19.803229 19455 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:05:19.803380 19455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "fdd14aac1ee04cbe90d86845fb5ae7de"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "dff16a5d7c2db19304153699f54ce39b"
server_key_iv: "acf5edd665aab3281e280f6d4129d119"
server_key_version: "encryptionkey@0"
I20260620 06:05:19.804016 19455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:19.825879 19455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:19.829262 19455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:19.829490 19455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:19.837357 19455 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42915
I20260620 06:05:19.837661 19523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42915 every 8 connection(s)
I20260620 06:05:19.838577 19455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:05:19.843046 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19455
I20260620 06:05:19.843209 19524 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:05:19.843629 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:35603
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:42915
--webserver_port=33755
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:05:19.845110 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap starting.
I20260620 06:05:19.869529 19524 log.cc:824] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:19.881153 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:05:19.881636 19524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap complete.
I20260620 06:05:19.887210 19524 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:19.887820 19524 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd14aac1ee04cbe90d86845fb5ae7de, State: Initialized, Role: FOLLOWER
I20260620 06:05:19.888453 19524 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:19.888613 19524 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:05:19.888705 19524 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:05:19.888810 19524 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 1 FOLLOWER]: Advancing to term 2
I20260620 06:05:19.891537 19524 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:19.891940 19524 leader_election.cc:302] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd14aac1ee04cbe90d86845fb5ae7de; no voters:
I20260620 06:05:19.892320 19524 leader_election.cc:288] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 06:05:19.892511 19528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Leader election won for term 2
I20260620 06:05:19.892892 19528 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 LEADER]: Becoming Leader. State: Replica: fdd14aac1ee04cbe90d86845fb5ae7de, State: Running, Role: LEADER
I20260620 06:05:19.893296 19528 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:19.893675 19524 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:05:19.894768 19530 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd14aac1ee04cbe90d86845fb5ae7de. Latest consensus state: current_term: 2 leader_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } } }
I20260620 06:05:19.894897 19530 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:19.895571 19529 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } } }
I20260620 06:05:19.895637 19537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:05:19.895658 19529 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:19.900058 19537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:05:19.900599 19537 catalog_manager.cc:1259] Loaded cluster ID: b793f2929f8342aea69716102d28fd8a
I20260620 06:05:19.900696 19537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:05:19.902482 19537 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:05:19.903134 19537 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Loaded TSK: 0
I20260620 06:05:19.926913 19537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Generated new TSK 1
I20260620 06:05:19.927237 19537 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:05:19.990222 19526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:19.990566 19526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:19.990670 19526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:19.995218 19526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:19.995339 19526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:19.995438 19526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:05:20.001117 19526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:35603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=33755
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:20.002236 19526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:20.003288 19526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:20.013989 19553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:20.014073 19552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:20.014211 19526 server_base.cc:1034] running on GCE node
W20260620 06:05:20.014581 19555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:20.015034 19526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:20.015756 19526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:20.017001 19526 hybrid_clock.cc:648] HybridClock initialized: now 1781935520016960 us; error 60 us; skew 500 ppm
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:20.020231 19526 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:05:20.021441 19526 webserver.cc:460] Webserver started at http://127.3.115.1:33755/ using document root <none> and password file <none>
I20260620 06:05:20.022096 19526 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:20.022179 19526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:20.026675 19526 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:05:20.029146 19562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.030613 19526 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:05:20.030728 19526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "81559fa0905cd4a11b06ff9debec9bc9"
server_key_iv: "152ccd56c1fc9c7ee2df21b5adfec528"
server_key_version: "encryptionkey@0"
I20260620 06:05:20.031455 19526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:20.051468 19526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:20.055365 19526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:20.055611 19526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:20.056314 19526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:20.057614 19569 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:05:20.062124 19526 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:05:20.062212 19526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 06:05:20.062328 19526 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:05:20.064229 19526 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:05:20.064321 19526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:05:20.064728 19569 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap starting.
I20260620 06:05:20.075630 19526 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35603
I20260620 06:05:20.075716 19676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35603 every 8 connection(s)
I20260620 06:05:20.076834 19526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:05:20.081977 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19526
I20260620 06:05:20.082521 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:37415
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:42915
--webserver_port=45043
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:05:20.085397 19569 log.cc:824] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:05:20.095623 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.079312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:44803 (local address 127.3.115.62:42915)
0620 06:05:20.079772 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.079798 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.080123 (+ 325us) server_negotiation.cc:407] Connection header received
0620 06:05:20.081407 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.081418 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.081758 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.082127 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:20.083742 (+ 1615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.085350 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.087009 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.087490 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.087675 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.090301 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:20.090307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:20.090322 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:20.090352 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:20.092115 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.092696 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.092708 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.092712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.092840 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.093190 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.093195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.093197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.093564 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:20.093791 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.094435 (+ 644us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.094758 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":129,"threads_started":1}
I20260620 06:05:20.095634 19680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.079304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:44803)
0620 06:05:20.079921 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.079978 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.080942 (+ 964us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.082183 (+ 1241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.082192 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.082966 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:20.083530 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.083546 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.085572 (+ 2026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.085576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.086809 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.086820 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.087643 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.087647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.087855 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.088529 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:20.088546 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:20.090076 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:20.092256 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.092262 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.092275 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.092544 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.092949 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.092952 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.092954 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.093093 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.093805 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:20.093813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:20.094051 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.094275 (+ 224us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.094626 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":125,"threads_started":1}
I20260620 06:05:20.095873 19569 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:05:20.096340 19569 tablet_bootstrap.cc:492] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap complete.
I20260620 06:05:20.097846 19569 ts_tablet_manager.cc:1397] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.015s
I20260620 06:05:20.098418 19677 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42915
I20260620 06:05:20.098672 19677 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:20.099174 19677 heartbeater.cc:511] Master 127.3.115.62:42915 requested a full tablet report, sending...
I20260620 06:05:20.100831 19569 raft_consensus.cc:348] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.101366 19569 raft_consensus.cc:729] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49a72e9b33044ed80bd0f6efba7d6ed, State: Initialized, Role: FOLLOWER
I20260620 06:05:20.101589 19488 ts_manager.cc:194] Registered new tserver with Master: d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603)
I20260620 06:05:20.101876 19569 consensus_queue.cc:260] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.102952 19569 ts_tablet_manager.cc:1428] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260620 06:05:20.105165 19488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:44803
I20260620 06:05:20.107872 19677 heartbeater.cc:503] Master 127.3.115.62:42915 was elected leader, sending a full tablet report...
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935519, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.1@KRBTEST.COM
I20260620 06:05:20.116108 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.104480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35603 (local address 127.0.0.1:50512)
0620 06:05:20.105375 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.105426 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.105574 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.106170 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.106182 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.106647 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:20.106795 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.106803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.108051 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.108055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.109116 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.109127 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.109969 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.109972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.110057 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.110689 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:20.110701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:20.112165 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:20.114155 (+ 1990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.114164 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.114170 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.114477 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.114817 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.114820 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.114822 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.114923 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.115427 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:20.115435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:20.115571 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.115720 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.115926 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":104,"threads_started":1}
I20260620 06:05:20.116139 19685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.104620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50512 (local address 127.3.115.1:35603)
0620 06:05:20.105041 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.105053 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.105534 (+ 481us) server_negotiation.cc:407] Connection header received
0620 06:05:20.105724 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.105729 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.105942 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.106116 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:20.106934 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.107934 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.109358 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.109844 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.109945 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.112336 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:20.112345 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:20.112355 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:20.112377 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:20.114043 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.114611 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.114623 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.114625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.114710 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.115020 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.115023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.115024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.115304 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:20.115411 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.115848 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.116012 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":139,"threads_started":1}
I20260620 06:05:20.117355 19611 tablet_service.cc:1514] Processing DeleteTablet for tablet d9df53a433014dfaa9490865581d60c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:04:59 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:50512
I20260620 06:05:20.118321 19686 tablet_replica.cc:330] stopping tablet replica
I20260620 06:05:20.118649 19686 raft_consensus.cc:2229] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:05:20.118928 19686 raft_consensus.cc:2258] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:05:20.120097 19686 ts_tablet_manager.cc:1905] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:05:20.124109 19686 ts_tablet_manager.cc:1918] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:05:20.124203 19686 log.cc:1196] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal/wals/d9df53a433014dfaa9490865581d60c7
I20260620 06:05:20.124506 19686 ts_tablet_manager.cc:1939] T d9df53a433014dfaa9490865581d60c7 P d49a72e9b33044ed80bd0f6efba7d6ed: Deleting consensus metadata
I20260620 06:05:20.126427 19473 catalog_manager.cc:4841] TS d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603): tablet d9df53a433014dfaa9490865581d60c7 (table test-table [id=e8764352fdd14f709396d31f5a1a4106]) successfully deleted
W20260620 06:05:20.207233 19681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:20.207566 19681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:20.207645 19681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:20.211982 19681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:20.212077 19681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:20.212172 19681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:05:20.217535 19681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:37415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=45043
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:20.218659 19681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:20.219774 19681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:20.229698 19692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:20.229810 19694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:20.229692 19691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:20.230011 19681 server_base.cc:1034] running on GCE node
I20260620 06:05:20.230489 19681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:20.231144 19681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:20.232328 19681 hybrid_clock.cc:648] HybridClock initialized: now 1781935520232291 us; error 49 us; skew 500 ppm
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:20.235345 19681 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:05:20.236423 19681 webserver.cc:460] Webserver started at http://127.3.115.2:45043/ using document root <none> and password file <none>
I20260620 06:05:20.237069 19681 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:20.237146 19681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:20.241218 19681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:05:20.243410 19701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.244910 19681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:05:20.245056 19681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "4b039a93c2844b5babea8061b4eda7e3"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "75be4115556fb285d5d15b138ce911d7"
server_key_iv: "50fd3c6171d0c92f8eca57bf0a1e6c57"
server_key_version: "encryptionkey@0"
I20260620 06:05:20.245636 19681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:20.273725 19681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:20.277307 19681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:20.277532 19681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:20.278205 19681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:20.279114 19681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:20.279187 19681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.279259 19681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:20.279294 19681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.288880 19681 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:37415
I20260620 06:05:20.288903 19814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:37415 every 8 connection(s)
I20260620 06:05:20.289939 19681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:05:20.290589 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19681
I20260620 06:05:20.291056 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:40497
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:42915
--webserver_port=37909
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:05:20.305335 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.291976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:39041 (local address 127.3.115.62:42915)
0620 06:05:20.292148 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.292152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.293252 (+ 1100us) server_negotiation.cc:407] Connection header received
0620 06:05:20.294829 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.294836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.294902 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.295006 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:20.296238 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.296880 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.298078 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.298334 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.298453 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.301396 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:20.301400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:20.301403 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:20.301437 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:20.303072 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.303655 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.303659 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.303661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.303748 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.304115 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.304120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.304122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.304317 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:20.304427 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.305024 (+ 597us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.305166 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:05:20.306229 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.292439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:39041)
0620 06:05:20.293022 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.293088 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.294564 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.295137 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.295143 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.295525 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:20.296078 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.296092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.297016 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.297020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.297942 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.297952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.298494 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.298498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.298677 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.299369 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:20.299386 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:20.301200 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:20.303213 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.303220 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.303238 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.303540 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.303852 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.303864 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.303866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.304014 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.304434 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:20.304443 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:20.304774 (+ 331us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.304985 (+ 211us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.305337 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":138,"threads_started":1}
I20260620 06:05:20.307533 19815 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42915
I20260620 06:05:20.307817 19815 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:20.308475 19815 heartbeater.cc:511] Master 127.3.115.62:42915 requested a full tablet report, sending...
I20260620 06:05:20.309767 19488 ts_manager.cc:194] Registered new tserver with Master: 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415)
I20260620 06:05:20.310529 19488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:39041
W20260620 06:05:20.410923 19817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:20.411227 19817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:20.411274 19817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:20.415697 19817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:20.415786 19817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:20.415849 19817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:05:20.421577 19817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:40497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=37909
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:20.422832 19817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:20.423945 19817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:20.434618 19824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:20.434653 19826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:20.434841 19817 server_base.cc:1034] running on GCE node
W20260620 06:05:20.434618 19823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:20.435390 19817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:20.436246 19817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:20.437428 19817 hybrid_clock.cc:648] HybridClock initialized: now 1781935520437396 us; error 53 us; skew 500 ppm
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:20.440387 19817 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:05:20.441341 19817 webserver.cc:460] Webserver started at http://127.3.115.3:37909/ using document root <none> and password file <none>
I20260620 06:05:20.441941 19817 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:20.442020 19817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:20.446014 19817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:05:20.448145 19833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.449622 19817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:05:20.449764 19817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "b5f6ee5332ef453db565c39700cb40f7"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "5b6d04775ee6efa54f0d6eaa59dc7495"
server_key_iv: "d1b2cbb02b47c99dbb921373f44d750f"
server_key_version: "encryptionkey@0"
I20260620 06:05:20.450382 19817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:20.463235 19817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:20.466678 19817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:20.466848 19817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:20.467489 19817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:20.468274 19817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:20.468322 19817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.468362 19817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:20.468377 19817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:20.477422 19817 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40497
I20260620 06:05:20.477442 19946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40497 every 8 connection(s)
I20260620 06:05:20.478489 19817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:05:20.479485 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19817
I20260620 06:05:20.489323 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.480249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34506 (local address 127.3.115.62:42915)
0620 06:05:20.480401 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.480406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.480834 (+ 428us) server_negotiation.cc:407] Connection header received
0620 06:05:20.480941 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.480944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.480997 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.481096 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:20.482010 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.482547 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.483724 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.483938 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.484014 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.485577 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:20.485580 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:20.485583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:20.485610 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:20.487397 (+ 1787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.487896 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.487899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.487901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.488036 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.488351 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.488353 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.488355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.488534 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:20.488765 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.488938 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.489076 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
Jun 20 06:05:20 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935520, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:05:20.493961 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.480401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:60615 (local address 127.3.115.62:42915)
0620 06:05:20.480599 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.480604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.481796 (+ 1192us) server_negotiation.cc:407] Connection header received
0620 06:05:20.483418 (+ 1622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.483420 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.483462 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.483532 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:20.485106 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.485677 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.486878 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.487259 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.487370 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.490352 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:20.490355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:20.490357 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:20.490377 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:20.491842 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.492358 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.492366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.492368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.492436 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.492858 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.492862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.492864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.493036 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:20.493139 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.493660 (+ 521us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.493777 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260620 06:05:20.494904 19951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.481012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:60615)
0620 06:05:20.481608 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.481672 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.483181 (+ 1509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.483740 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.483750 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.484217 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:20.484943 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.484966 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.485800 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.485803 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.486745 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.486755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.487412 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.487415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.487636 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.488295 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:20.488322 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:20.490173 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:20.491963 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.491968 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.491984 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.492248 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.492535 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:20.492538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:20.492540 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:20.492753 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:20.493152 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:20.493161 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:20.493413 (+ 252us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.493606 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.493980 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":145,"threads_started":1}
I20260620 06:05:20.495973 19947 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42915
I20260620 06:05:20.496237 19947 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:20.496819 19947 heartbeater.cc:511] Master 127.3.115.62:42915 requested a full tablet report, sending...
I20260620 06:05:20.497908 19488 ts_manager.cc:194] Registered new tserver with Master: b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
I20260620 06:05:20.498497 19488 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:60615
I20260620 06:05:20.503050 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:05:20.507221 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.503694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34508 (local address 127.3.115.62:42915)
0620 06:05:20.503861 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.503864 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.503966 (+ 102us) server_negotiation.cc:407] Connection header received
0620 06:05:20.504071 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.504074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.504199 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.504320 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:05:20.504540 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.505128 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.506034 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.506304 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.506387 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.506693 (+ 306us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:05:20.506828 (+ 135us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 06:05:20.507036 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34508: token expired
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 06:05:20.507305 19950 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34508: token expired
W20260620 06:05:20.507606 19415 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.115.62:42915)
I20260620 06:05:20.515539 19950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.508012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.3.115.62:42915)
0620 06:05:20.508129 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.508136 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.508206 (+ 70us) server_negotiation.cc:407] Connection header received
0620 06:05:20.508385 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.508388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.508438 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.508518 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:20.509323 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.509822 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.510656 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.510887 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.510996 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.512341 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:20.512345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:20.512347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:20.512374 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:20.513825 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.514260 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.514265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.514267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.514351 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:20.514662 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:20.514667 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:20.514668 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:20.514866 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:20.515010 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.515244 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.515363 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:05:20.518100 19488 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34524:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:05:20.521564 19488 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:05:20.530153 19611 tablet_service.cc:1467] Processing CreateTablet for tablet ca51338b20cc4e1782f435c6f56cd42a (DEFAULT_TABLE table=test-table [id=a22753a701b8431ea419abbe9033741c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:05:20.530920 19611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca51338b20cc4e1782f435c6f56cd42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:20.534057 19569 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap starting.
I20260620 06:05:20.534529 19684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.528771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:37415 (local address 127.0.0.1:51716)
0620 06:05:20.528869 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.528886 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.528965 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.529990 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.529992 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.530011 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:20.530133 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.530140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.531556 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.531563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.532830 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.532842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.534043 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.534047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.534164 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.534213 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.534267 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.534354 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 06:05:20.535058 19569 tablet_bootstrap.cc:654] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:20.535236 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.529118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40497 (local address 127.0.0.1:45860)
0620 06:05:20.529363 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.529377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.529465 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.530555 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.530559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.530578 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:20.530665 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.530671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.532049 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.532055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.533221 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.533231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.534531 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.534535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.534895 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.534934 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.535004 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.535090 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":122,"threads_started":1}
I20260620 06:05:20.535595 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.529039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51716 (local address 127.3.115.2:37415)
0620 06:05:20.529450 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.529463 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.529488 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:05:20.529577 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.529583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.529837 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.530031 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:20.530234 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.531425 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.532993 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.533896 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.535059 (+ 1163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.535232 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.535309 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.535418 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":176,"threads_started":1}
I20260620 06:05:20.535859 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.529619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45860 (local address 127.3.115.3:40497)
0620 06:05:20.529991 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.530004 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.530028 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:05:20.530105 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.530110 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.530316 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.530474 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:20.530814 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.531915 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.533346 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.533962 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.534081 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.535476 (+ 1395us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.535545 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.535651 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":160,"threads_started":1}
I20260620 06:05:20.536819 19569 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: No bootstrap required, opened a new log
I20260620 06:05:20.536918 19569 ts_tablet_manager.cc:1397] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260620 06:05:20.537365 19569 raft_consensus.cc:348] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.537499 19569 raft_consensus.cc:374] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:20.537542 19569 raft_consensus.cc:729] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49a72e9b33044ed80bd0f6efba7d6ed, State: Initialized, Role: FOLLOWER
I20260620 06:05:20.537658 19569 consensus_queue.cc:260] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.537909 19569 ts_tablet_manager.cc:1428] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 06:05:20.538992 19749 tablet_service.cc:1467] Processing CreateTablet for tablet ca51338b20cc4e1782f435c6f56cd42a (DEFAULT_TABLE table=test-table [id=a22753a701b8431ea419abbe9033741c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:05:20.539258 19881 tablet_service.cc:1467] Processing CreateTablet for tablet ca51338b20cc4e1782f435c6f56cd42a (DEFAULT_TABLE table=test-table [id=a22753a701b8431ea419abbe9033741c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:05:20.539958 19749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca51338b20cc4e1782f435c6f56cd42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:20.540175 19881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca51338b20cc4e1782f435c6f56cd42a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:20.547689 19960 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Bootstrap starting.
I20260620 06:05:20.547747 19961 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Bootstrap starting.
I20260620 06:05:20.550350 19961 tablet_bootstrap.cc:654] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:20.550459 19960 tablet_bootstrap.cc:654] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:20.551213 19961 log.cc:824] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:20.551210 19960 log.cc:824] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:20.553140 19960 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: No bootstrap required, opened a new log
I20260620 06:05:20.553150 19961 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: No bootstrap required, opened a new log
I20260620 06:05:20.553352 19960 ts_tablet_manager.cc:1397] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 06:05:20.553361 19961 ts_tablet_manager.cc:1397] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260620 06:05:20.556756 19961 raft_consensus.cc:348] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.556758 19960 raft_consensus.cc:348] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.557031 19961 raft_consensus.cc:374] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:20.557027 19960 raft_consensus.cc:374] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:20.557092 19961 raft_consensus.cc:729] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5f6ee5332ef453db565c39700cb40f7, State: Initialized, Role: FOLLOWER
I20260620 06:05:20.557099 19960 raft_consensus.cc:729] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b039a93c2844b5babea8061b4eda7e3, State: Initialized, Role: FOLLOWER
I20260620 06:05:20.557575 19960 consensus_queue.cc:260] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.557574 19961 consensus_queue.cc:260] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.558337 19815 heartbeater.cc:503] Master 127.3.115.62:42915 was elected leader, sending a full tablet report...
I20260620 06:05:20.558455 19947 heartbeater.cc:503] Master 127.3.115.62:42915 was elected leader, sending a full tablet report...
I20260620 06:05:20.558692 19960 ts_tablet_manager.cc:1428] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:05:20.559856 19961 ts_tablet_manager.cc:1428] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 06:05:20.578114 19678 tablet.cc:2369] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:05:20.668146 19965 raft_consensus.cc:482] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:20.668367 19965 raft_consensus.cc:504] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.669607 19965 leader_election.cc:288] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497), d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603)
I20260620 06:05:20.673590 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.670033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:38723 (local address 127.3.115.3:40497)
0620 06:05:20.670193 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.670198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.670218 (+ 20us) server_negotiation.cc:407] Connection header received
0620 06:05:20.670367 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.670370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.670452 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.670523 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:20.670939 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.671452 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.672535 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.673025 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.673145 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.673220 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.673402 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.673474 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:05:20.673590 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.669866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40497 (local address 127.3.115.2:38723)
0620 06:05:20.670072 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.670091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.670212 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.670653 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.670658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.670678 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:20.670823 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.670829 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.671581 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.671584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.672401 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.672411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.673146 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.673149 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.673260 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.673289 (+ 29us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.673374 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.673450 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 06:05:20.673879 19966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.670188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35603 (local address 127.3.115.2:60909)
0620 06:05:20.670585 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.670598 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:20.670692 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:20.670983 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:20.670987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:20.671004 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:20.671134 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.671139 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.671946 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.671952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.672774 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:20.672783 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.673482 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.673486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:20.673576 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.673596 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:05:20.673657 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 06:05:20.673734 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":155,"threads_started":1}
I20260620 06:05:20.673906 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.670363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:60909 (local address 127.3.115.1:35603)
0620 06:05:20.670599 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.670603 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.670706 (+ 103us) server_negotiation.cc:407] Connection header received
0620 06:05:20.670768 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.670770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.670827 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.670972 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:20.671291 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.671813 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.672904 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.673355 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.673481 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.673661 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.673699 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.673769 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":84,"threads_started":1}
I20260620 06:05:20.674376 19901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca51338b20cc4e1782f435c6f56cd42a" candidate_uuid: "4b039a93c2844b5babea8061b4eda7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f6ee5332ef453db565c39700cb40f7" is_pre_election: true
I20260620 06:05:20.674733 19901 raft_consensus.cc:2454] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b039a93c2844b5babea8061b4eda7e3 in term 0.
I20260620 06:05:20.674779 19631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca51338b20cc4e1782f435c6f56cd42a" candidate_uuid: "4b039a93c2844b5babea8061b4eda7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" is_pre_election: true
I20260620 06:05:20.675102 19631 raft_consensus.cc:2454] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b039a93c2844b5babea8061b4eda7e3 in term 0.
I20260620 06:05:20.675278 19703 leader_election.cc:302] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7; no voters:
I20260620 06:05:20.675606 19965 raft_consensus.cc:2781] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:05:20.675698 19965 raft_consensus.cc:482] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:05:20.675743 19965 raft_consensus.cc:3037] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:20.676887 19965 raft_consensus.cc:504] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.677232 19965 leader_election.cc:288] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [CANDIDATE]: Term 1 election: Requested vote from peers b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497), d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603)
I20260620 06:05:20.677600 19901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca51338b20cc4e1782f435c6f56cd42a" candidate_uuid: "4b039a93c2844b5babea8061b4eda7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5f6ee5332ef453db565c39700cb40f7"
I20260620 06:05:20.677706 19631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca51338b20cc4e1782f435c6f56cd42a" candidate_uuid: "4b039a93c2844b5babea8061b4eda7e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
I20260620 06:05:20.677747 19901 raft_consensus.cc:3037] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:20.677875 19631 raft_consensus.cc:3037] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:20.678875 19631 raft_consensus.cc:2454] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b039a93c2844b5babea8061b4eda7e3 in term 1.
I20260620 06:05:20.678871 19901 raft_consensus.cc:2454] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b039a93c2844b5babea8061b4eda7e3 in term 1.
I20260620 06:05:20.679163 19703 leader_election.cc:302] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b039a93c2844b5babea8061b4eda7e3, d49a72e9b33044ed80bd0f6efba7d6ed; no voters:
I20260620 06:05:20.679347 19965 raft_consensus.cc:2781] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:05:20.679600 19965 raft_consensus.cc:686] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 4b039a93c2844b5babea8061b4eda7e3, State: Running, Role: LEADER
I20260620 06:05:20.679940 19965 consensus_queue.cc:237] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:20.683195 19488 catalog_manager.cc:5495] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2). New cstate: current_term: 1 leader_uuid: "4b039a93c2844b5babea8061b4eda7e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } health_report { overall_health: UNKNOWN } } }
I20260620 06:05:20.698599 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.695022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51720 (local address 127.3.115.2:37415)
0620 06:05:20.695154 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:20.695158 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:20.695171 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:05:20.695280 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:20.695282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:20.695338 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:20.695475 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:05:20.695712 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.696225 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.697138 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:20.697348 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:20.697466 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:20.697729 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:05:20.698152 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:05:20.698246 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:20.698416 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:05:20.698469 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:05:20.704952 19631 raft_consensus.cc:1261] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Refusing update from remote peer 4b039a93c2844b5babea8061b4eda7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:05:20.704962 19901 raft_consensus.cc:1261] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Refusing update from remote peer 4b039a93c2844b5babea8061b4eda7e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:05:20.705706 19968 consensus_queue.cc:1035] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:05:20.705979 19965 consensus_queue.cc:1035] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:05:20.713135 19972 mvcc.cc:205] Tried to move back new op lower bound from 7298807892799143936 to 7298807892708052992. Current Snapshot: MvccSnapshot[applied={T|T < 7298807892799143936}]
I20260620 06:05:20.718204 19974 mvcc.cc:205] Tried to move back new op lower bound from 7298807892799143936 to 7298807892708052992. Current Snapshot: MvccSnapshot[applied={T|T < 7298807892799143936}]
I20260620 06:05:20.722306 19487 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:05:20.722509 19487 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34524:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:05:20.725210 19487 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Sending DeleteTablet for 3 replicas of tablet ca51338b20cc4e1782f435c6f56cd42a
I20260620 06:05:20.726006 19749 tablet_service.cc:1514] Processing DeleteTablet for tablet ca51338b20cc4e1782f435c6f56cd42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:05:20 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:51716
I20260620 06:05:20.726212 19611 tablet_service.cc:1514] Processing DeleteTablet for tablet ca51338b20cc4e1782f435c6f56cd42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:05:20 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:50512
I20260620 06:05:20.726280 19881 tablet_service.cc:1514] Processing DeleteTablet for tablet ca51338b20cc4e1782f435c6f56cd42a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:05:20 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:45860
I20260620 06:05:20.726804 19982 tablet_replica.cc:330] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: stopping tablet replica
I20260620 06:05:20.726817 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19526
I20260620 06:05:20.728732 19981 tablet_replica.cc:330] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: stopping tablet replica
I20260620 06:05:20.729100 19981 raft_consensus.cc:2229] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:05:20.729101 19982 raft_consensus.cc:2229] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:05:20.729485 19982 raft_consensus.cc:2258] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:05:20.729591 19981 raft_consensus.cc:2258] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:05:20.731276 19981 ts_tablet_manager.cc:1905] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:05:20.731276 19982 ts_tablet_manager.cc:1905] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:05:20.734289 19982 ts_tablet_manager.cc:1918] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:05:20.734289 19981 ts_tablet_manager.cc:1918] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:05:20.734401 19981 log.cc:1196] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/ca51338b20cc4e1782f435c6f56cd42a
I20260620 06:05:20.734529 19982 log.cc:1196] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/ca51338b20cc4e1782f435c6f56cd42a
W20260620 06:05:20.734503 19473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:35603 (error 108)
I20260620 06:05:20.734797 19981 ts_tablet_manager.cc:1939] T ca51338b20cc4e1782f435c6f56cd42a P 4b039a93c2844b5babea8061b4eda7e3: Deleting consensus metadata
I20260620 06:05:20.734948 19982 ts_tablet_manager.cc:1939] T ca51338b20cc4e1782f435c6f56cd42a P b5f6ee5332ef453db565c39700cb40f7: Deleting consensus metadata
I20260620 06:05:20.736179 19475 catalog_manager.cc:4841] TS 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415): tablet ca51338b20cc4e1782f435c6f56cd42a (table test-table [id=a22753a701b8431ea419abbe9033741c]) successfully deleted
I20260620 06:05:20.736183 19473 catalog_manager.cc:4841] TS b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497): tablet ca51338b20cc4e1782f435c6f56cd42a (table test-table [id=a22753a701b8431ea419abbe9033741c]) successfully deleted
I20260620 06:05:20.736441 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:20.736035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35603 (local address 127.0.0.1:50520)
0620 06:05:20.736284 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:20.736354 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35603: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156}
I20260620 06:05:20.736883 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19681
W20260620 06:05:20.736613 19473 catalog_manager.cc:4584] TS d49a72e9b33044ed80bd0f6efba7d6ed (127.3.115.1:35603): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ca51338b20cc4e1782f435c6f56cd42a: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35603: connect: Connection refused (error 111)
I20260620 06:05:20.742595 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19817
I20260620 06:05:20.748342 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 19455
I20260620 06:05:40.755313 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42915
--webserver_interface=127.3.115.62
--webserver_port=38463
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:05:40.869100 19984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:40.869390 19984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:40.869431 19984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:40.873543 19984 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 06:05:40.873607 19984 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 06:05:40.873624 19984 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 06:05:40.873637 19984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:05:40.873656 19984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:40.873674 19984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:40.873689 19984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:05:40.873705 19984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:05:40.879168 19984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42915
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=38463
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:40.880260 19984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:40.881378 19984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:40.890105 19990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:40.890038 19989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:40.890012 19992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:40.890466 19984 server_base.cc:1034] running on GCE node
I20260620 06:05:40.890962 19984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:40.892084 19984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:40.893293 19984 hybrid_clock.cc:648] HybridClock initialized: now 1781935540893259 us; error 49 us; skew 500 ppm
Jun 20 06:05:40 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935540, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:40.896214 19984 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:05:40.897264 19984 webserver.cc:460] Webserver started at http://127.3.115.62:38463/ using document root <none> and password file <none>
I20260620 06:05:40.897853 19984 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:40.897955 19984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:40.901973 19984 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:05:40.904260 19999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:40.905726 19984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:05:40.905871 19984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "fdd14aac1ee04cbe90d86845fb5ae7de"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "dff16a5d7c2db19304153699f54ce39b"
server_key_iv: "acf5edd665aab3281e280f6d4129d119"
server_key_version: "encryptionkey@0"
I20260620 06:05:40.906481 19984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:40.921927 19984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:40.925454 19984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:40.925681 19984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:40.933398 19984 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42915
I20260620 06:05:40.933688 20051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42915 every 8 connection(s)
I20260620 06:05:40.934638 19984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:05:40.938913 20052 sys_catalog.cc:263] Verifying existing consensus state
I20260620 06:05:40.940563 20052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap starting.
I20260620 06:05:40.942811 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 19984
I20260620 06:05:40.943289 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:35603
--local_ip_for_outbound_sockets=127.3.115.1
--tserver_master_addrs=127.3.115.62:42915
--webserver_port=33755
--webserver_interface=127.3.115.1
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:05:40.958810 20052 log.cc:824] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:40.971045 20052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:05:40.971467 20052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Bootstrap complete.
I20260620 06:05:40.975613 20052 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:40.976027 20052 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd14aac1ee04cbe90d86845fb5ae7de, State: Initialized, Role: FOLLOWER
I20260620 06:05:40.976529 20052 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:40.976643 20052 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:05:40.976758 20052 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:05:40.976881 20052 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 2 FOLLOWER]: Advancing to term 3
I20260620 06:05:40.978919 20052 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:40.979267 20052 leader_election.cc:302] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdd14aac1ee04cbe90d86845fb5ae7de; no voters:
I20260620 06:05:40.979641 20052 leader_election.cc:288] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 06:05:40.979730 20056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 3 FOLLOWER]: Leader election won for term 3
I20260620 06:05:40.979933 20056 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [term 3 LEADER]: Becoming Leader. State: Replica: fdd14aac1ee04cbe90d86845fb5ae7de, State: Running, Role: LEADER
I20260620 06:05:40.980296 20056 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } }
I20260620 06:05:40.980957 20052 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:05:40.981730 20057 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } } }
I20260620 06:05:40.981866 20057 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:40.982227 20058 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdd14aac1ee04cbe90d86845fb5ae7de. Latest consensus state: current_term: 3 leader_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdd14aac1ee04cbe90d86845fb5ae7de" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42915 } } }
I20260620 06:05:40.982357 20058 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:40.982774 20063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:05:40.988047 20063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:05:40.988746 20063 catalog_manager.cc:1259] Loaded cluster ID: b793f2929f8342aea69716102d28fd8a
I20260620 06:05:40.988806 20063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:05:40.991360 20063 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:05:40.992125 20063 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Loaded TSK: 0
I20260620 06:05:40.992210 20063 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fdd14aac1ee04cbe90d86845fb5ae7de: Loaded TSK: 1
F20260620 06:05:40.993047 20063 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8ea3c919ad google::LogMessage::Fail() at ??:0
@ 0x7f8ea3c9488a google::LogMessage::SendToLog() at ??:0
@ 0x7f8ea3c914de google::LogMessage::Flush() at ??:0
@ 0x7f8ea3c92a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8ea695518d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8ea694df0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8eae5f67bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8eae5d099c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8eae5d1178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f8eae612df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8eae63182b std::function<>::operator()() at ??:0
@ 0x7f8eae5d0b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f8eae5d1f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8eae5ce579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f8eae612640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8ea5ee33d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8ea5ee3c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f8ea5ee53f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f8ea5ed4512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8ea75186db start_thread at ??:0
@ 0x7f8ea278f61f clone at ??:0
W20260620 06:05:41.096454 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:41.096818 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:41.096897 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:41.101087 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:41.101147 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:41.101233 20054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:05:41.106760 20054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:35603
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=33755
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:41.107893 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:41.109014 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:41.119995 20100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:41.120158 20054 server_base.cc:1034] running on GCE node
W20260620 06:05:41.120028 20099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:41.120375 20102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:41.120846 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:41.121536 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:41.122725 20054 hybrid_clock.cc:648] HybridClock initialized: now 1781935541122707 us; error 35 us; skew 500 ppm
Jun 20 06:05:41 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935541, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:41.125571 20054 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:05:41.126555 20054 webserver.cc:460] Webserver started at http://127.3.115.1:33755/ using document root <none> and password file <none>
I20260620 06:05:41.127135 20054 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:41.127197 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:41.131237 20054 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:05:41.133459 20109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.134774 20054 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:05:41.134903 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "d49a72e9b33044ed80bd0f6efba7d6ed"
format_stamp: "Formatted at 2026-06-20 06:04:58 on dist-test-slave-4h14"
server_key: "81559fa0905cd4a11b06ff9debec9bc9"
server_key_iv: "152ccd56c1fc9c7ee2df21b5adfec528"
server_key_version: "encryptionkey@0"
I20260620 06:05:41.135445 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:41.154628 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:41.158290 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:41.158524 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:41.159231 20054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:41.160418 20116 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 06:05:41.164744 20054 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 06:05:41.164817 20054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260620 06:05:41.164901 20054 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 06:05:41.166745 20054 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 06:05:41.166805 20054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260620 06:05:41.167068 20116 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap starting.
I20260620 06:05:41.177812 20054 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35603
I20260620 06:05:41.177901 20223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35603 every 8 connection(s)
I20260620 06:05:41.179201 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:05:41.181166 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20054
I20260620 06:05:41.181644 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:37415
--local_ip_for_outbound_sockets=127.3.115.2
--tserver_master_addrs=127.3.115.62:42915
--webserver_port=45043
--webserver_interface=127.3.115.2
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:05:41.188618 20116 log.cc:824] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:41.199664 20116 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 06:05:41.200191 20116 tablet_bootstrap.cc:492] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Bootstrap complete.
I20260620 06:05:41.201746 20116 ts_tablet_manager.cc:1397] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.007s
I20260620 06:05:41.205348 20116 raft_consensus.cc:348] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:41.206120 20116 raft_consensus.cc:729] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d49a72e9b33044ed80bd0f6efba7d6ed, State: Initialized, Role: FOLLOWER
I20260620 06:05:41.206743 20116 consensus_queue.cc:260] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:41.207945 20116 ts_tablet_manager.cc:1428] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260620 06:05:41.219391 20226 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:05:41.181675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:36673)
0620 06:05:41.183077 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.183138 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:41.184426 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:41.218596 (+ 34170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1158,"thread_start_us":188,"threads_started":1}
W20260620 06:05:41.220067 20224 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:42915 (0 consecutive failures): Network error: Failed to ping master at 127.3.115.62:42915: Client connection negotiation failed: client connection to 127.3.115.62:42915: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 06:05:41.220973 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.220632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:45641)
0620 06:05:41.220805 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.220885 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260620 06:05:41.221758 20226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.221500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:52589)
0620 06:05:41.221631 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.221692 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 06:05:41.222002 20224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:05:41.313956 20227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:41.314250 20227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:41.314297 20227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:41.319051 20227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:41.319126 20227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:41.319222 20227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:05:41.324908 20227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:37415
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=45043
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:41.326135 20227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:41.327350 20227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:05:41.337850 20227 server_base.cc:1034] running on GCE node
W20260620 06:05:41.337806 20237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:41.337903 20234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:41.337813 20235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:41.338503 20227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:41.339181 20227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:41.340343 20227 hybrid_clock.cc:648] HybridClock initialized: now 1781935541340298 us; error 60 us; skew 500 ppm
Jun 20 06:05:41 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935541, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:41.343189 20227 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:05:41.344079 20227 webserver.cc:460] Webserver started at http://127.3.115.2:45043/ using document root <none> and password file <none>
I20260620 06:05:41.344621 20227 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:41.344751 20227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:41.348784 20227 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:05:41.350975 20244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.352314 20227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:05:41.352413 20227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "4b039a93c2844b5babea8061b4eda7e3"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "75be4115556fb285d5d15b138ce911d7"
server_key_iv: "50fd3c6171d0c92f8eca57bf0a1e6c57"
server_key_version: "encryptionkey@0"
I20260620 06:05:41.353067 20227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:41.377406 20227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:41.380929 20227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:41.381109 20227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:41.381691 20227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:41.382550 20227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:41.382596 20227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.382637 20227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:41.382652 20227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.392220 20227 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:37415
I20260620 06:05:41.392256 20357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:37415 every 8 connection(s)
I20260620 06:05:41.393290 20227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:05:41.394210 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20227
I20260620 06:05:41.394709 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:40497
--local_ip_for_outbound_sockets=127.3.115.3
--tserver_master_addrs=127.3.115.62:42915
--webserver_port=37909
--webserver_interface=127.3.115.3
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:39757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:05:41.398247 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.395986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:46903)
0620 06:05:41.396545 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.397269 (+ 724us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
W20260620 06:05:41.398748 20358 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:42915 (0 consecutive failures): Network error: Failed to ping master at 127.3.115.62:42915: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
I20260620 06:05:41.399461 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.399145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:40825)
0620 06:05:41.399296 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.399366 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 06:05:41.400209 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.399938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:52209)
0620 06:05:41.400070 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.400134 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:05:41.400403 20358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 06:05:41.510273 20360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:41.510560 20360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:41.510686 20360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:41.514784 20360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:41.514853 20360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:41.514918 20360 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:05:41.520358 20360 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:39757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:40497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=37909
--tserver_master_addrs=127.3.115.62:42915
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:41.521561 20360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:41.522563 20360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:41.533034 20369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:41.533025 20367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:41.533025 20366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:41.533079 20360 server_base.cc:1034] running on GCE node
I20260620 06:05:41.533667 20360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:41.534384 20360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:41.535604 20360 hybrid_clock.cc:648] HybridClock initialized: now 1781935541535576 us; error 45 us; skew 500 ppm
Jun 20 06:05:41 dist-test-slave-4h14 krb5kdc[18906](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935541, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:41.538749 20360 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:05:41.539786 20360 webserver.cc:460] Webserver started at http://127.3.115.3:37909/ using document root <none> and password file <none>
I20260620 06:05:41.540431 20360 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:41.540535 20360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:41.544899 20360 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:05:41.547253 20376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.548640 20360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:05:41.548869 20360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "b5f6ee5332ef453db565c39700cb40f7"
format_stamp: "Formatted at 2026-06-20 06:04:59 on dist-test-slave-4h14"
server_key: "5b6d04775ee6efa54f0d6eaa59dc7495"
server_key_iv: "d1b2cbb02b47c99dbb921373f44d750f"
server_key_version: "encryptionkey@0"
I20260620 06:05:41.549510 20360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:41.562633 20360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:41.566039 20360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:41.566327 20360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:41.566946 20360 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:41.567780 20360 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:41.567857 20360 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.567934 20360 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:41.567989 20360 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:41.577124 20360 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40497
I20260620 06:05:41.577153 20489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40497 every 8 connection(s)
I20260620 06:05:41.578135 20360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:05:41.581920 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20360
I20260620 06:05:41.582271 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.580304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:38489)
0620 06:05:41.580837 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.581345 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":104,"threads_started":1}
W20260620 06:05:41.582772 20490 heartbeater.cc:644] Failed to heartbeat to 127.3.115.62:42915 (0 consecutive failures): Network error: Failed to ping master at 127.3.115.62:42915: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
I20260620 06:05:41.583489 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.583210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:59917)
0620 06:05:41.583336 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.583390 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:05:41.584120 20492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:41.583931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:49849)
0620 06:05:41.584029 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:41.584072 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:05:41.584322 20490 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 06:05:42.223031 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.222603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:33697)
0620 06:05:42.222856 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:42.222914 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
I20260620 06:05:42.401466 20495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.401034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:35211)
0620 06:05:42.401292 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:42.401352 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260620 06:05:42.443754 20496 raft_consensus.cc:482] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:42.443971 20496 raft_consensus.cc:504] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:42.445200 20496 leader_election.cc:288] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
Jun 20 06:05:42 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935541, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.3@KRBTEST.COM
Jun 20 06:05:42 dist-test-slave-4h14 krb5kdc[18906](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935541, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.2@KRBTEST.COM
I20260620 06:05:42.459580 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.445440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40497 (local address 127.3.115.1:46177)
0620 06:05:42.445563 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:42.445579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:42.445689 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:42.447400 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:42.447406 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:42.447956 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:42.448436 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:42.448449 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.449789 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.449793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:42.450761 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:42.450770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.451473 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.451477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:42.451733 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:42.452503 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:42.452519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:42.454415 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:42.457078 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:42.457081 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:42.457083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:42.457384 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:42.457820 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:42.457823 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:42.457824 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:42.457932 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:42.458551 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:42.458557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:42.458800 (+ 243us) client_negotiation.cc:769] Sending connection context
0620 06:05:42.459001 (+ 201us) client_negotiation.cc:240] Negotiation successful
0620 06:05:42.459398 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":212,"spinlock_wait_cycles":27520}
I20260620 06:05:42.459595 20497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.445557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:47315 (local address 127.3.115.2:37415)
0620 06:05:42.445935 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:42.445967 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:42.446276 (+ 309us) server_negotiation.cc:407] Connection header received
0620 06:05:42.447058 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:42.447063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:42.447424 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:42.447714 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:42.448613 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.449650 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.450890 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.451355 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.451529 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:42.454988 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:42.454994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:42.455008 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:42.455039 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:42.456871 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:42.457467 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:42.457477 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:42.457483 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:42.457630 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:42.457978 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:42.457984 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:42.457987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:42.458431 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:42.458581 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:42.459111 (+ 530us) server_negotiation.cc:299] Negotiation successful
0620 06:05:42.459389 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":115,"threads_started":1}
I20260620 06:05:42.459595 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.445647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:46177 (local address 127.3.115.3:40497)
0620 06:05:42.445956 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:42.445966 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:42.446001 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:05:42.446658 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:42.446664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:42.447030 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:42.447274 (+ 244us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:42.448618 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.449639 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.450898 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.451350 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.451508 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:42.454671 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:42.454676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:42.454690 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:42.454721 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:42.456959 (+ 2238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:42.457528 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:42.457535 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:42.457543 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:42.457717 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:42.458030 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:42.458040 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:42.458042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:42.458430 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:42.458563 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:42.459109 (+ 546us) server_negotiation.cc:299] Negotiation successful
0620 06:05:42.459407 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":78,"threads_started":1}
I20260620 06:05:42.459678 20499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.445438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:37415 (local address 127.3.115.1:47315)
0620 06:05:42.446140 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:42.446151 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:42.446285 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:42.447675 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:42.447677 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:42.447955 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:42.448435 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:42.448450 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.449806 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.449809 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:42.450759 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:42.450768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:42.451496 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:42.451500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:42.451736 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:42.452558 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:42.452582 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:42.454784 (+ 2202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:42.456988 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:42.456997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:42.457013 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:42.457348 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:42.457705 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:42.457708 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:42.457710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:42.457907 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:42.458564 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:42.458568 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:42.458808 (+ 240us) client_negotiation.cc:769] Sending connection context
0620 06:05:42.459000 (+ 192us) client_negotiation.cc:240] Negotiation successful
0620 06:05:42.459485 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"mutex_wait_us":750,"spinlock_wait_cycles":9856,"thread_start_us":71,"threads_started":1}
W20260620 06:05:42.462198 20113 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
W20260620 06:05:42.462376 20111 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
I20260620 06:05:42.462497 20111 leader_election.cc:302] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d49a72e9b33044ed80bd0f6efba7d6ed; no voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7
I20260620 06:05:42.462792 20496 raft_consensus.cc:2726] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:05:42.585330 20500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:42.584925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:34805)
0620 06:05:42.585180 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:42.585237 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
I20260620 06:05:43.224453 20501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:43.224004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:56149)
0620 06:05:43.224287 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:43.224348 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":77,"threads_started":1}
I20260620 06:05:43.402820 20502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:43.402365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:55271)
0620 06:05:43.402666 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:43.402726 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260620 06:05:43.586721 20503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:43.586268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:39285)
0620 06:05:43.586545 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:43.586608 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260620 06:05:44.225718 20504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:44.225297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:44553)
0620 06:05:44.225555 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:44.225616 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":79,"threads_started":1}
I20260620 06:05:44.404219 20505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:44.403715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:53851)
0620 06:05:44.404020 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:44.404100 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":76,"threads_started":1}
I20260620 06:05:44.425917 20506 raft_consensus.cc:482] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:44.426098 20506 raft_consensus.cc:504] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:44.426515 20506 leader_election.cc:288] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
W20260620 06:05:44.427310 20113 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
W20260620 06:05:44.427449 20111 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
I20260620 06:05:44.427537 20111 leader_election.cc:302] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d49a72e9b33044ed80bd0f6efba7d6ed; no voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7
I20260620 06:05:44.427726 20506 raft_consensus.cc:2726] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:05:44.588064 20507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:44.587551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:52087)
0620 06:05:44.587863 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:44.587921 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":115,"threads_started":1}
I20260620 06:05:45.227113 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:45.226642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:54739)
0620 06:05:45.226931 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:45.227000 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":80,"threads_started":1}
I20260620 06:05:45.405753 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:45.405244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:44023)
0620 06:05:45.405567 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:45.405629 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":118,"threads_started":1}
I20260620 06:05:45.589574 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:45.589040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:60419)
0620 06:05:45.589374 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:45.589446 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":86,"threads_started":1}
I20260620 06:05:46.228865 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:46.228258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:51525)
0620 06:05:46.228630 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:46.228764 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":97,"threads_started":1}
I20260620 06:05:46.407305 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:46.406774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:54413)
0620 06:05:46.407134 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:46.407203 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260620 06:05:46.591059 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:46.590582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:51173)
0620 06:05:46.590893 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:46.590959 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":84,"threads_started":1}
I20260620 06:05:47.230202 20514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:47.229699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:60475)
0620 06:05:47.230033 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:47.230102 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":98,"threads_started":1}
I20260620 06:05:47.408905 20515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:47.408231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:46609)
0620 06:05:47.408574 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:47.408643 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":97,"threads_started":1}
I20260620 06:05:47.592374 20516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:47.591890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:33593)
0620 06:05:47.592206 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:47.592271 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":82,"threads_started":1}
I20260620 06:05:47.620797 20517 raft_consensus.cc:482] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:47.620926 20517 raft_consensus.cc:504] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:47.621345 20517 leader_election.cc:288] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
W20260620 06:05:47.622092 20111 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
W20260620 06:05:47.622184 20113 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
I20260620 06:05:47.622305 20113 leader_election.cc:302] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d49a72e9b33044ed80bd0f6efba7d6ed; no voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7
I20260620 06:05:47.622459 20517 raft_consensus.cc:2726] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:05:48.231371 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:48.230986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:42585)
0620 06:05:48.231222 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:48.231275 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":69,"threads_started":1}
I20260620 06:05:48.410303 20519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:48.409772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:40285)
0620 06:05:48.410134 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:48.410202 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":89,"threads_started":1}
I20260620 06:05:48.594082 20520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:48.593438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:40145)
0620 06:05:48.593912 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:48.593971 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":90,"threads_started":1}
I20260620 06:05:49.232932 20521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:49.232321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:39687)
0620 06:05:49.232608 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:49.232753 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":83,"threads_started":1}
I20260620 06:05:49.411777 20522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:49.411252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:41509)
0620 06:05:49.411604 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:49.411668 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":128,"threads_started":1}
I20260620 06:05:49.595402 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:49.594908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:39987)
0620 06:05:49.595176 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:49.595275 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":78,"threads_started":1}
I20260620 06:05:50.234287 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:50.233782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:50135)
0620 06:05:50.234129 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:50.234192 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":94,"threads_started":1}
I20260620 06:05:50.342111 20525 raft_consensus.cc:482] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:50.342248 20525 raft_consensus.cc:504] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b039a93c2844b5babea8061b4eda7e3" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 37415 } } peers { permanent_uuid: "b5f6ee5332ef453db565c39700cb40f7" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40497 } } peers { permanent_uuid: "d49a72e9b33044ed80bd0f6efba7d6ed" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35603 } }
I20260620 06:05:50.342746 20525 leader_election.cc:288] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415), b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497)
W20260620 06:05:50.343590 20113 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4b039a93c2844b5babea8061b4eda7e3 (127.3.115.2:37415): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
W20260620 06:05:50.343712 20111 leader_election.cc:341] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5f6ee5332ef453db565c39700cb40f7 (127.3.115.3:40497): Not found: Tablet not found: ca51338b20cc4e1782f435c6f56cd42a
I20260620 06:05:50.343850 20111 leader_election.cc:302] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d49a72e9b33044ed80bd0f6efba7d6ed; no voters: 4b039a93c2844b5babea8061b4eda7e3, b5f6ee5332ef453db565c39700cb40f7
I20260620 06:05:50.344003 20525 raft_consensus.cc:2726] T ca51338b20cc4e1782f435c6f56cd42a P d49a72e9b33044ed80bd0f6efba7d6ed [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 06:05:50.413458 20526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:50.412938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:37849)
0620 06:05:50.413251 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:50.413350 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":89,"threads_started":1}
I20260620 06:05:50.596808 20527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:50.596255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:40219)
0620 06:05:50.596571 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:50.596638 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":86,"threads_started":1}
I20260620 06:05:51.235666 20528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:51.235220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:47451)
0620 06:05:51.235501 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:51.235569 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
I20260620 06:05:51.415104 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:51.414558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:53305)
0620 06:05:51.414911 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:51.414999 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":78,"threads_started":1}
I20260620 06:05:51.598120 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:51.597646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:44143)
0620 06:05:51.597963 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:51.598025 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":71,"threads_started":1}
I20260620 06:05:52.237058 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:52.236512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:44075)
0620 06:05:52.236894 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:52.236960 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":93,"threads_started":1}
I20260620 06:05:52.416776 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:52.416095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:50513)
0620 06:05:52.416427 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:52.416518 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":101,"threads_started":1}
I20260620 06:05:52.599426 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:52.598986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:60577)
0620 06:05:52.599260 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:52.599325 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
I20260620 06:05:53.238507 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:53.238011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:49691)
0620 06:05:53.238348 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:53.238409 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":91,"threads_started":1}
I20260620 06:05:53.418252 20535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:53.417748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:59937)
0620 06:05:53.418075 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:53.418147 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":85,"threads_started":1}
I20260620 06:05:53.600952 20536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:53.600350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:45007)
0620 06:05:53.600753 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:53.600812 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":113,"threads_started":1}
I20260620 06:05:54.239722 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:54.239280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:45199)
0620 06:05:54.239554 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:54.239612 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":75,"threads_started":1}
I20260620 06:05:54.419787 20538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:54.419260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:49695)
0620 06:05:54.419612 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:54.419682 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":88,"threads_started":1}
I20260620 06:05:54.602180 20539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:54.601732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:46907)
0620 06:05:54.602026 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:54.602082 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":91,"threads_started":1}
I20260620 06:05:55.241137 20540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:55.240624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:46505)
0620 06:05:55.240977 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:55.241039 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":107,"threads_started":1}
I20260620 06:05:55.421231 20541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:55.420771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:47263)
0620 06:05:55.421063 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:55.421129 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":75,"threads_started":1}
I20260620 06:05:55.603433 20542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:55.602979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.3:58785)
0620 06:05:55.603261 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:55.603322 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":91,"threads_started":1}
I20260620 06:05:56.242568 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:56.242125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.1:54033)
0620 06:05:56.242399 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:56.242469 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":84,"threads_started":1}
I20260620 06:05:56.422668 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:56.422234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42915 (local address 127.3.115.2:47167)
0620 06:05:56.422489 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:56.422562 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.62:42915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260620 06:05:56.590025 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20054
I20260620 06:05:56.596087 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20227
I20260620 06:05:56.601558 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20360
2026-06-20T06:05:56Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59387 ms)
[----------] 30 tests from SecurityITest (245059 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:05:56 dist-test-slave-4h14 krb5kdc[20547](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:05:56 dist-test-slave-4h14 krb5kdc[20547](info): set up 2 sockets
Jun 20 06:05:56 dist-test-slave-4h14 krb5kdc[20547](info): commencing operation
krb5kdc: starting...
W20260620 06:05:57.644585 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:05:57 dist-test-slave-4h14 krb5kdc[20547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:05:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:05:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:05:57.763705 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:41853
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:41853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:05:57.874147 20560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:57.874485 20560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:57.874562 20560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:57.878607 20560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:05:57.878700 20560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:57.878741 20560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:57.878798 20560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:05:57.878839 20560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:05:57.884166 20560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:41853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:41853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:57.885322 20560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:57.886380 20560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:05:57.895139 20560 server_base.cc:1034] running on GCE node
W20260620 06:05:57.895193 20565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:57.895191 20566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:57.895191 20568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:57.895972 20560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:57.897167 20560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:57.898362 20560 hybrid_clock.cc:648] HybridClock initialized: now 1781935557898344 us; error 32 us; skew 500 ppm
Jun 20 06:05:57 dist-test-slave-4h14 krb5kdc[20547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935557, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:57.901214 20560 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:05:57.902148 20560 webserver.cc:460] Webserver started at http://127.3.115.62:43793/ using document root <none> and password file <none>
I20260620 06:05:57.902706 20560 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:57.902796 20560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:57.903025 20560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:57.905148 20560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93"
format_stamp: "Formatted at 2026-06-20 06:05:57 on dist-test-slave-4h14"
server_key: "d3f733d91183e8fd4c1c6c0b2a54cf38"
server_key_iv: "b4704674278237a7e9e6a27394f50960"
server_key_version: "encryptionkey@0"
I20260620 06:05:57.905726 20560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93"
format_stamp: "Formatted at 2026-06-20 06:05:57 on dist-test-slave-4h14"
server_key: "d3f733d91183e8fd4c1c6c0b2a54cf38"
server_key_iv: "b4704674278237a7e9e6a27394f50960"
server_key_version: "encryptionkey@0"
I20260620 06:05:57.909521 20560 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:05:57.912070 20575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:57.913544 20560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:05:57.913705 20560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93"
format_stamp: "Formatted at 2026-06-20 06:05:57 on dist-test-slave-4h14"
server_key: "d3f733d91183e8fd4c1c6c0b2a54cf38"
server_key_iv: "b4704674278237a7e9e6a27394f50960"
server_key_version: "encryptionkey@0"
I20260620 06:05:57.913826 20560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:57.934599 20560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:57.937604 20560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:57.937808 20560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:57.945173 20560 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:41853
I20260620 06:05:57.945199 20627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:41853 every 8 connection(s)
I20260620 06:05:57.946154 20560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:05:57.949249 20628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:57.949945 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20560
I20260620 06:05:57.950089 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:05:57.950460 3532 external_mini_cluster.cc:1428] Setting key f9dd19f33ba9c2d766364621007ee512
I20260620 06:05:57.955533 20628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93: Bootstrap starting.
I20260620 06:05:57.958400 20628 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:57.959165 20628 log.cc:824] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:05:57 dist-test-slave-4h14 krb5kdc[20547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:05:57.961961 20628 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93: No bootstrap required, opened a new log
I20260620 06:05:57.965446 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:57.951918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.3.115.62:41853)
0620 06:05:57.952370 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:57.952386 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:57.952424 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:05:57.953143 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:57.953162 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:57.953510 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:57.953880 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:57.954665 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:57.955689 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:57.956989 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:57.957477 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:57.957651 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:57.960517 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:57.960522 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:57.960537 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:57.960564 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:57.962528 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:57.963003 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:57.963009 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:57.963013 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:57.963129 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:57.963421 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:57.963424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:57.963426 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:57.963821 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:57.964040 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:57.964332 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 06:05:57.964628 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260620 06:05:57.965698 20628 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41853 } }
I20260620 06:05:57.965975 20628 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:57.966058 20628 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c3a2d7d8cff4cb6bfefa56d62f14b93, State: Initialized, Role: FOLLOWER
I20260620 06:05:57.966573 20628 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41853 } }
I20260620 06:05:57.966737 20628 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:05:57.966802 20628 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:05:57.966897 20628 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:57.968081 20628 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41853 } }
I20260620 06:05:57.968384 20628 leader_election.cc:302] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5c3a2d7d8cff4cb6bfefa56d62f14b93; no voters:
I20260620 06:05:57.968642 20628 leader_election.cc:288] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:05:57.968809 20633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:05:57.969065 20633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [term 1 LEADER]: Becoming Leader. State: Replica: 5c3a2d7d8cff4cb6bfefa56d62f14b93, State: Running, Role: LEADER
I20260620 06:05:57.969415 20633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41853 } }
I20260620 06:05:57.969911 20628 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:05:57.971078 20634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41853 } } }
I20260620 06:05:57.971218 20634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:57.971592 20642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:05:57.971580 20635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5c3a2d7d8cff4cb6bfefa56d62f14b93. Latest consensus state: current_term: 1 leader_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5c3a2d7d8cff4cb6bfefa56d62f14b93" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41853 } } }
I20260620 06:05:57.971735 20635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93 [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:57.974821 20642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:05:57.980458 20642 catalog_manager.cc:1347] Generated new cluster ID: e01074bf1c7d47aaba8a10cdafe82a38
I20260620 06:05:57.980528 20642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:05:57.994681 20642 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:05:57.995669 20642 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:05:58.000257 20642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93: Generated new TSK 0
I20260620 06:05:58.001044 20642 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:05:58.029949 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41853
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:05:58.139699 20654 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:58.139984 20654 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:58.140054 20654 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:58.144044 20654 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:58.144142 20654 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:58.144235 20654 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:05:58.149503 20654 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20654
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:58.150620 20654 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:58.151688 20654 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:58.161329 20662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:58.161336 20659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:58.161391 20654 server_base.cc:1034] running on GCE node
W20260620 06:05:58.161336 20660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:58.162030 20654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:58.162707 20654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:58.163872 20654 hybrid_clock.cc:648] HybridClock initialized: now 1781935558163852 us; error 42 us; skew 500 ppm
Jun 20 06:05:58 dist-test-slave-4h14 krb5kdc[20547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:58.166885 20654 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:05:58.167754 20654 webserver.cc:460] Webserver started at http://127.3.115.1:43051/ using document root <none> and password file <none>
I20260620 06:05:58.168308 20654 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:58.168368 20654 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:58.168534 20654 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:58.170747 20654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "1bffdd3a50d542d9b600da2547811617"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "b8599b13ab24263e423e90fecf5a8e3b"
server_key_iv: "d0db2c5446d61139bfdcd21ff1c76e2a"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.171342 20654 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "1bffdd3a50d542d9b600da2547811617"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "b8599b13ab24263e423e90fecf5a8e3b"
server_key_iv: "d0db2c5446d61139bfdcd21ff1c76e2a"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.175072 20654 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260620 06:05:58.177592 20669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.178896 20654 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260620 06:05:58.179057 20654 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "1bffdd3a50d542d9b600da2547811617"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "b8599b13ab24263e423e90fecf5a8e3b"
server_key_iv: "d0db2c5446d61139bfdcd21ff1c76e2a"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.179174 20654 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:58.189224 20654 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:58.192728 20654 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:58.192898 20654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:58.193459 20654 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:58.194255 20654 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:58.194301 20654 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.194396 20654 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:58.194419 20654 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.203143 20654 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:38439
I20260620 06:05:58.203169 20782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:38439 every 8 connection(s)
I20260620 06:05:58.204088 20654 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:05:58.206550 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20654
I20260620 06:05:58.206707 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:05:58.207058 3532 external_mini_cluster.cc:1428] Setting key 9273b139810e0c146814bad4e570a411
Jun 20 06:05:58 dist-test-slave-4h14 krb5kdc[20547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:05:58.219097 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.205991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37681 (local address 127.3.115.62:41853)
0620 06:05:58.206175 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.206180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.207212 (+ 1032us) server_negotiation.cc:407] Connection header received
0620 06:05:58.208363 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.208366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.208399 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.208540 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:58.210175 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.211026 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.212198 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.212447 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.212557 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.215098 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:58.215102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:58.215104 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:58.215129 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:58.216919 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.217437 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.217441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.217443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.217544 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.217918 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.217923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.217924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.218160 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:58.218270 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.218766 (+ 496us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.218916 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:05:58.220180 20785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.206450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:41853 (local address 127.3.115.1:37681)
0620 06:05:58.207025 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.207087 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.208158 (+ 1071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.208753 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.208762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.209290 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:58.210013 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.210036 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.211167 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.211169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.212066 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.212075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.212607 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.212611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.212876 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.213484 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:58.213500 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:58.214928 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:58.217033 (+ 2105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:58.217039 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:58.217055 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:58.217321 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:58.217657 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:58.217660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:58.217662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:58.217822 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:58.218267 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:58.218273 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:58.218530 (+ 257us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.218726 (+ 196us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.219076 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":133,"threads_started":1}
I20260620 06:05:58.221529 20783 heartbeater.cc:347] Connected to a master server at 127.3.115.62:41853
I20260620 06:05:58.221817 20783 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:58.222443 20783 heartbeater.cc:511] Master 127.3.115.62:41853 requested a full tablet report, sending...
I20260620 06:05:58.224031 20592 ts_manager.cc:194] Registered new tserver with Master: 1bffdd3a50d542d9b600da2547811617 (127.3.115.1:38439)
I20260620 06:05:58.225329 20592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:37681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:05:58.233664 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41853
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:05:58.347308 20788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:58.347605 20788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:58.347651 20788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:58.351780 20788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:58.351848 20788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:58.351949 20788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:05:58.357296 20788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:58.358441 20788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:58.359453 20788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:58.369059 20794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:58.369053 20796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:58.369102 20788 server_base.cc:1034] running on GCE node
W20260620 06:05:58.369053 20793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:58.369737 20788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:58.370399 20788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:58.371563 20788 hybrid_clock.cc:648] HybridClock initialized: now 1781935558371532 us; error 45 us; skew 500 ppm
Jun 20 06:05:58 dist-test-slave-4h14 krb5kdc[20547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:58.374302 20788 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:05:58.375149 20788 webserver.cc:460] Webserver started at http://127.3.115.2:38003/ using document root <none> and password file <none>
I20260620 06:05:58.375697 20788 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:58.375756 20788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:58.375957 20788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:58.377995 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "ce3f09c6f4e344368d20da4080ed32bd"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "f44e7f456affa540c86d57cd8a6f077c"
server_key_iv: "ca1585da83a8b48bf729818a01d432a1"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.378543 20788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "ce3f09c6f4e344368d20da4080ed32bd"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "f44e7f456affa540c86d57cd8a6f077c"
server_key_iv: "ca1585da83a8b48bf729818a01d432a1"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.382157 20788 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:05:58.384553 20803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.385911 20788 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260620 06:05:58.386039 20788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "ce3f09c6f4e344368d20da4080ed32bd"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "f44e7f456affa540c86d57cd8a6f077c"
server_key_iv: "ca1585da83a8b48bf729818a01d432a1"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.386142 20788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:58.403251 20788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:58.406641 20788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:58.406848 20788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:58.407465 20788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:58.408291 20788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:58.408339 20788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.408377 20788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:58.408443 20788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.418228 20788 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:35575
I20260620 06:05:58.418269 20916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:35575 every 8 connection(s)
I20260620 06:05:58.419209 20788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:05:58.421274 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20788
I20260620 06:05:58.421430 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:05:58.421737 3532 external_mini_cluster.cc:1428] Setting key de64556f40d58f6ae2477de7a0452d56
Jun 20 06:05:58 dist-test-slave-4h14 krb5kdc[20547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:05:58.434664 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.421089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:38247 (local address 127.3.115.62:41853)
0620 06:05:58.421238 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.421243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.422291 (+ 1048us) server_negotiation.cc:407] Connection header received
0620 06:05:58.423702 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.423708 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.423772 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.423866 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:58.425398 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.425931 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.427782 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.428007 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.428088 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.431066 (+ 2978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:58.431069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:58.431072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:58.431096 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:58.432587 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.433121 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.433127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.433129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.433211 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.433548 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.433554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.433558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.433744 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:58.433847 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.434370 (+ 523us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.434501 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:05:58.435604 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.421651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:41853 (local address 127.3.115.2:38247)
0620 06:05:58.422135 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.422191 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.423493 (+ 1302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.424020 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.424027 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.424382 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:58.425181 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.425205 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.426069 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.426076 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.427652 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.427665 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.428169 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.428173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.428430 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.429174 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:58.429201 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:58.430879 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:58.432725 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:58.432731 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:58.432748 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:58.433017 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:58.433299 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:58.433303 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:58.433305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:58.433449 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:58.433845 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:58.433852 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:58.434128 (+ 276us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.434328 (+ 200us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.434675 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":142,"threads_started":1}
I20260620 06:05:58.436874 20917 heartbeater.cc:347] Connected to a master server at 127.3.115.62:41853
I20260620 06:05:58.437129 20917 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:58.437613 20917 heartbeater.cc:511] Master 127.3.115.62:41853 requested a full tablet report, sending...
I20260620 06:05:58.438692 20592 ts_manager.cc:194] Registered new tserver with Master: ce3f09c6f4e344368d20da4080ed32bd (127.3.115.2:35575)
I20260620 06:05:58.439604 20592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:38247
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:05:58.449362 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41853
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:38915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:05:58.560948 20922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:58.561216 20922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:58.561259 20922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:58.565320 20922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:58.565382 20922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:58.565442 20922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:05:58.570869 20922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:38915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:58.572003 20922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:58.573112 20922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:58.582684 20927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:58.582686 20930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:58.582814 20928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:58.582933 20922 server_base.cc:1034] running on GCE node
I20260620 06:05:58.583477 20922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:58.584173 20922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:58.585343 20922 hybrid_clock.cc:648] HybridClock initialized: now 1781935558585323 us; error 32 us; skew 500 ppm
Jun 20 06:05:58 dist-test-slave-4h14 krb5kdc[20547](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:05:58.588106 20922 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:05:58.589116 20922 webserver.cc:460] Webserver started at http://127.3.115.3:44949/ using document root <none> and password file <none>
I20260620 06:05:58.589736 20922 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:58.589824 20922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:58.590111 20922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:58.592226 20922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "d11e16d778798f2ca70687b2965af35e"
server_key_iv: "7d86510e3618a6ff3572ddc362b48cf2"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.592854 20922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "d11e16d778798f2ca70687b2965af35e"
server_key_iv: "7d86510e3618a6ff3572ddc362b48cf2"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.596537 20922 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:05:58.599037 20937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.600293 20922 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:05:58.600456 20922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a"
format_stamp: "Formatted at 2026-06-20 06:05:58 on dist-test-slave-4h14"
server_key: "d11e16d778798f2ca70687b2965af35e"
server_key_iv: "7d86510e3618a6ff3572ddc362b48cf2"
server_key_version: "encryptionkey@0"
I20260620 06:05:58.600579 20922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:58.613607 20922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:58.617105 20922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:58.617378 20922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:58.618033 20922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:58.618938 20922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:58.619015 20922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.619094 20922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:58.619139 20922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:58.628225 20922 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:43859
I20260620 06:05:58.628244 21050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:43859 every 8 connection(s)
I20260620 06:05:58.629331 20922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:05:58.635648 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 20922
I20260620 06:05:58.635792 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:05:58.636081 3532 external_mini_cluster.cc:1428] Setting key fb343cfd5253a5068d2cad98bc70d974
Jun 20 06:05:58 dist-test-slave-4h14 krb5kdc[20547](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935558, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:05:58.643769 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.631343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:55853 (local address 127.3.115.62:41853)
0620 06:05:58.631541 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.631545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.632413 (+ 868us) server_negotiation.cc:407] Connection header received
0620 06:05:58.633823 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.633826 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.633884 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.633973 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:58.635146 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.635714 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.637128 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.637453 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.637585 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.640328 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:58.640331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:58.640333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:58.640392 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:58.641917 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.642407 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.642409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.642411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.642486 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.642768 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.642770 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.642771 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.642940 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:58.643017 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.643536 (+ 519us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.643641 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260620 06:05:58.644639 21053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.631752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:41853 (local address 127.3.115.3:55853)
0620 06:05:58.632251 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.632297 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.633605 (+ 1308us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.634142 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.634149 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.634513 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:58.635001 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.635015 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.635871 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.635877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.636979 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.636989 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.637646 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.637651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.637868 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.638607 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:05:58.638622 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:58.640148 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:58.642031 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:58.642035 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:58.642047 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:58.642320 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:58.642561 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:05:58.642563 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:05:58.642564 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:05:58.642695 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:05:58.643047 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:58.643053 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:58.643289 (+ 236us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.643469 (+ 180us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.643749 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":118,"threads_started":1}
I20260620 06:05:58.645929 21051 heartbeater.cc:347] Connected to a master server at 127.3.115.62:41853
I20260620 06:05:58.646186 21051 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:58.646723 21051 heartbeater.cc:511] Master 127.3.115.62:41853 requested a full tablet report, sending...
I20260620 06:05:58.647842 20592 ts_manager.cc:194] Registered new tserver with Master: 9cc2ec1bdbd94b3a8bfcf51b35d0198a (127.3.115.3:43859)
I20260620 06:05:58.648458 20592 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:55853
I20260620 06:05:58.650228 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:05:58.659304 20631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.652166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.3.115.62:41853)
0620 06:05:58.652275 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.652280 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.652435 (+ 155us) server_negotiation.cc:407] Connection header received
0620 06:05:58.652540 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.652542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.652582 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.652693 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:58.653363 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.653871 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.654639 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.654851 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.654936 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.656438 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:58.656441 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:58.656443 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:05:58.656464 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:58.657777 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.658181 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.658183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.658184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.658257 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:05:58.658482 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:05:58.658484 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:05:58.658485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:05:58.658697 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:58.658806 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.659078 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.659179 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:05:58.662350 20592 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:05:58.665503 20592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:05:58.680313 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.674166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:43859 (local address 127.0.0.1:54108)
0620 06:05:58.674557 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.674611 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.674751 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.675841 (+ 1090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.675846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.675869 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.676038 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.676045 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.677399 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.677403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.678845 (+ 1442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.678856 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.679734 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.679738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.679853 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.679893 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.680007 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.680128 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":115,"threads_started":1}
I20260620 06:05:58.680480 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.674301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.3.115.3:43859)
0620 06:05:58.675322 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.675334 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.675358 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:05:58.675447 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.675453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.675680 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.675856 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.676155 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.677246 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.678997 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.679611 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.679736 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.679915 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.680201 (+ 286us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.680310 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":138,"threads_started":1}
I20260620 06:05:58.680480 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.674840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38372 (local address 127.3.115.1:38439)
0620 06:05:58.675792 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.675803 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.675821 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:05:58.675884 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.675888 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.676030 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.676473 (+ 443us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.676632 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.677700 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.679134 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.679915 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.680013 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.680130 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.680254 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.680336 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":125,"threads_started":1}
I20260620 06:05:58.681001 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.675154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51808 (local address 127.3.115.2:35575)
0620 06:05:58.675673 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.675684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.675707 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:05:58.675781 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.675787 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.676007 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.676165 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.676562 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.677638 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.678949 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.679778 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.679888 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.680041 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.680742 (+ 701us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.680835 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":92,"threads_started":1}
I20260620 06:05:58.681247 21066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.674998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:35575 (local address 127.0.0.1:51808)
0620 06:05:58.675507 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.675520 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.675601 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.676324 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.676327 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.676346 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.676440 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.676446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.677791 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.677795 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.678818 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.678826 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.680106 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.680109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.680413 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.680437 (+ 24us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.680509 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.680596 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":241,"thread_start_us":97,"threads_started":1}
I20260620 06:05:58.681246 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.674456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:38439 (local address 127.0.0.1:38372)
0620 06:05:58.675311 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.675330 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.675417 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.676172 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.676175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.676191 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.676303 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.676312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.677845 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.677849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.679018 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.679026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.680023 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.680026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.680106 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.680162 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.680233 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.680293 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260620 06:05:58.684024 20985 tablet_service.cc:1467] Processing CreateTablet for tablet e4b310ec54224552ac1a2ff3d93b1b3d (DEFAULT_TABLE table=test-table [id=3e1f382242064aa6a1d1607104e522d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:05:58.684201 20717 tablet_service.cc:1467] Processing CreateTablet for tablet e4b310ec54224552ac1a2ff3d93b1b3d (DEFAULT_TABLE table=test-table [id=3e1f382242064aa6a1d1607104e522d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:05:58.684198 20851 tablet_service.cc:1467] Processing CreateTablet for tablet e4b310ec54224552ac1a2ff3d93b1b3d (DEFAULT_TABLE table=test-table [id=3e1f382242064aa6a1d1607104e522d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:05:58.685070 20985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b310ec54224552ac1a2ff3d93b1b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:58.685220 20717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b310ec54224552ac1a2ff3d93b1b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:58.685220 20851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4b310ec54224552ac1a2ff3d93b1b3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:58.692561 21068 tablet_bootstrap.cc:492] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Bootstrap starting.
I20260620 06:05:58.693341 21070 tablet_bootstrap.cc:492] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Bootstrap starting.
I20260620 06:05:58.693470 21069 tablet_bootstrap.cc:492] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: Bootstrap starting.
I20260620 06:05:58.695176 21068 tablet_bootstrap.cc:654] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:58.695495 21069 tablet_bootstrap.cc:654] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:58.695919 21070 tablet_bootstrap.cc:654] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:58.696803 21068 log.cc:824] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:58.696832 21070 log.cc:824] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:58.696832 21069 log.cc:824] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:58.698539 21069 tablet_bootstrap.cc:492] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: No bootstrap required, opened a new log
I20260620 06:05:58.698730 21070 tablet_bootstrap.cc:492] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: No bootstrap required, opened a new log
I20260620 06:05:58.698696 21068 tablet_bootstrap.cc:492] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: No bootstrap required, opened a new log
I20260620 06:05:58.698755 21069 ts_tablet_manager.cc:1397] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260620 06:05:58.698904 21068 ts_tablet_manager.cc:1397] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260620 06:05:58.698915 21070 ts_tablet_manager.cc:1397] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:05:58.701836 21069 raft_consensus.cc:348] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.702162 21069 raft_consensus.cc:374] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:58.702239 21069 raft_consensus.cc:729] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1bffdd3a50d542d9b600da2547811617, State: Initialized, Role: FOLLOWER
I20260620 06:05:58.702698 21069 consensus_queue.cc:260] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.702749 21068 raft_consensus.cc:348] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.703022 21068 raft_consensus.cc:374] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:58.703174 21068 raft_consensus.cc:729] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc2ec1bdbd94b3a8bfcf51b35d0198a, State: Initialized, Role: FOLLOWER
I20260620 06:05:58.703020 21070 raft_consensus.cc:348] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.703262 21070 raft_consensus.cc:374] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:58.703348 21070 raft_consensus.cc:729] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce3f09c6f4e344368d20da4080ed32bd, State: Initialized, Role: FOLLOWER
I20260620 06:05:58.703409 20783 heartbeater.cc:503] Master 127.3.115.62:41853 was elected leader, sending a full tablet report...
I20260620 06:05:58.703755 21069 ts_tablet_manager.cc:1428] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:05:58.703796 21068 consensus_queue.cc:260] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.703796 21070 consensus_queue.cc:260] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.704452 21051 heartbeater.cc:503] Master 127.3.115.62:41853 was elected leader, sending a full tablet report...
I20260620 06:05:58.704838 21070 ts_tablet_manager.cc:1428] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:05:58.704866 21068 ts_tablet_manager.cc:1428] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260620 06:05:58.705168 20917 heartbeater.cc:503] Master 127.3.115.62:41853 was elected leader, sending a full tablet report...
W20260620 06:05:58.705452 20784 tablet.cc:2369] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:05:58.880559 21052 tablet.cc:2369] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:05:58.920434 20918 tablet.cc:2369] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:05:58.955598 21074 raft_consensus.cc:482] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:58.955686 21076 raft_consensus.cc:482] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:05:58.955839 21074 raft_consensus.cc:504] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.955926 21076 raft_consensus.cc:504] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.957969 21076 leader_election.cc:288] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cc2ec1bdbd94b3a8bfcf51b35d0198a (127.3.115.3:43859), 1bffdd3a50d542d9b600da2547811617 (127.3.115.1:38439)
I20260620 06:05:58.958259 21074 leader_election.cc:288] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cc2ec1bdbd94b3a8bfcf51b35d0198a (127.3.115.3:43859), ce3f09c6f4e344368d20da4080ed32bd (127.3.115.2:35575)
I20260620 06:05:58.962675 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.958372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:40511 (local address 127.3.115.2:35575)
0620 06:05:58.958577 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.958581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.958594 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:05:58.958638 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.958640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.958676 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.958759 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.959187 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960012 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.961337 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.961973 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.962070 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.962132 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.962458 (+ 326us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.962538 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":140}
I20260620 06:05:58.962860 21079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:43859 (local address 127.3.115.2:43161)
0620 06:05:58.958096 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.958112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.958217 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.958849 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.958853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.958868 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.958977 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.958986 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960149 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.960152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.961608 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.961617 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.962430 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.962436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.962544 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.962567 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.962620 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.962704 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"mutex_wait_us":331,"thread_start_us":61,"threads_started":1}
I20260620 06:05:58.962984 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:41683 (local address 127.3.115.3:43859)
0620 06:05:58.957957 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.957960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.958459 (+ 499us) server_negotiation.cc:407] Connection header received
0620 06:05:58.958660 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.958663 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.958741 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.958786 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.959276 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960053 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.961609 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.962225 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.962316 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.962400 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.962747 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.962818 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":244,"spinlock_wait_cycles":41728}
I20260620 06:05:58.963424 21080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:43161 (local address 127.3.115.3:43859)
0620 06:05:58.958593 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.958602 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.958616 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:05:58.958658 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.958660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.958733 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.959082 (+ 349us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.959186 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960014 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.961724 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.962293 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.963125 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.963197 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.963231 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.963289 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":855,"spinlock_wait_cycles":2816,"thread_start_us":117,"threads_started":1}
I20260620 06:05:58.963552 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:60483 (local address 127.3.115.1:38439)
0620 06:05:58.957927 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:58.957932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:58.958149 (+ 217us) server_negotiation.cc:407] Connection header received
0620 06:05:58.958269 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:58.958274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:58.958326 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:58.958414 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:05:58.959275 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960053 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.962348 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.962937 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.963007 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.963053 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:58.963376 (+ 323us) server_negotiation.cc:299] Negotiation successful
0620 06:05:58.963435 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:05:58.963644 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:43859 (local address 127.3.115.1:41683)
0620 06:05:58.958370 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.958398 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.958532 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.958929 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.958931 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.958947 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.959147 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.959154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960194 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.960197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.961465 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.961476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.962489 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.962493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.962617 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.962638 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.963305 (+ 667us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.963377 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"mutex_wait_us":71,"thread_start_us":117,"threads_started":1}
I20260620 06:05:58.963478 21081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:35575 (local address 127.3.115.1:40511)
0620 06:05:58.958463 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.958476 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.958539 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.958879 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.958882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.958907 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.959063 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.959073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960161 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.960164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.961206 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.961215 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.962191 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.962195 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.962298 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.962336 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.963224 (+ 888us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.963306 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260620 06:05:58.963817 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:58.957413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:38439 (local address 127.3.115.2:60483)
0620 06:05:58.958028 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:58.958046 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:58.958162 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:58.959012 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:58.959017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:58.959036 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:05:58.959154 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.959161 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.960197 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.960199 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.962228 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:58.962236 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:58.963144 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:58.963148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:58.963251 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:58.963270 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 06:05:58.963506 (+ 236us) client_negotiation.cc:240] Negotiation successful
0620 06:05:58.963577 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":56,"thread_start_us":111,"threads_started":1}
I20260620 06:05:58.964128 21005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "ce3f09c6f4e344368d20da4080ed32bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" is_pre_election: true
I20260620 06:05:58.964445 20737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "ce3f09c6f4e344368d20da4080ed32bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bffdd3a50d542d9b600da2547811617" is_pre_election: true
I20260620 06:05:58.964445 20871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "1bffdd3a50d542d9b600da2547811617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3f09c6f4e344368d20da4080ed32bd" is_pre_election: true
I20260620 06:05:58.964478 21005 raft_consensus.cc:2454] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3f09c6f4e344368d20da4080ed32bd in term 0.
I20260620 06:05:58.964673 20871 raft_consensus.cc:2454] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bffdd3a50d542d9b600da2547811617 in term 0.
I20260620 06:05:58.964674 20737 raft_consensus.cc:2454] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce3f09c6f4e344368d20da4080ed32bd in term 0.
I20260620 06:05:58.965059 20807 leader_election.cc:302] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1bffdd3a50d542d9b600da2547811617, 9cc2ec1bdbd94b3a8bfcf51b35d0198a, ce3f09c6f4e344368d20da4080ed32bd; no voters:
I20260620 06:05:58.965062 20673 leader_election.cc:302] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1bffdd3a50d542d9b600da2547811617, ce3f09c6f4e344368d20da4080ed32bd; no voters:
I20260620 06:05:58.965355 21074 raft_consensus.cc:2781] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:05:58.965487 21074 raft_consensus.cc:482] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:05:58.965459 21005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "1bffdd3a50d542d9b600da2547811617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" is_pre_election: true
I20260620 06:05:58.965552 21074 raft_consensus.cc:3037] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:58.965651 21005 raft_consensus.cc:2454] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1bffdd3a50d542d9b600da2547811617 in term 0.
I20260620 06:05:58.965921 21076 raft_consensus.cc:2781] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:05:58.966053 21076 raft_consensus.cc:482] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:05:58.966128 21076 raft_consensus.cc:3037] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:58.966848 21074 raft_consensus.cc:504] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.967172 21076 raft_consensus.cc:504] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.967214 21074 leader_election.cc:288] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [CANDIDATE]: Term 1 election: Requested vote from peers 9cc2ec1bdbd94b3a8bfcf51b35d0198a (127.3.115.3:43859), ce3f09c6f4e344368d20da4080ed32bd (127.3.115.2:35575)
I20260620 06:05:58.967557 21005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "1bffdd3a50d542d9b600da2547811617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a"
I20260620 06:05:58.967557 20871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "1bffdd3a50d542d9b600da2547811617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce3f09c6f4e344368d20da4080ed32bd"
I20260620 06:05:58.967666 21005 raft_consensus.cc:3037] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:58.967725 20871 raft_consensus.cc:2379] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1bffdd3a50d542d9b600da2547811617 in current term 1: Already voted for candidate ce3f09c6f4e344368d20da4080ed32bd in this term.
I20260620 06:05:58.967824 21004 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "ce3f09c6f4e344368d20da4080ed32bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a"
I20260620 06:05:58.968288 20737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4b310ec54224552ac1a2ff3d93b1b3d" candidate_uuid: "ce3f09c6f4e344368d20da4080ed32bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1bffdd3a50d542d9b600da2547811617"
I20260620 06:05:58.968477 20737 raft_consensus.cc:2379] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ce3f09c6f4e344368d20da4080ed32bd in current term 1: Already voted for candidate 1bffdd3a50d542d9b600da2547811617 in this term.
I20260620 06:05:58.968770 21076 leader_election.cc:288] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [CANDIDATE]: Term 1 election: Requested vote from peers 9cc2ec1bdbd94b3a8bfcf51b35d0198a (127.3.115.3:43859), 1bffdd3a50d542d9b600da2547811617 (127.3.115.1:38439)
I20260620 06:05:58.968900 20804 leader_election.cc:302] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce3f09c6f4e344368d20da4080ed32bd; no voters: 1bffdd3a50d542d9b600da2547811617, 9cc2ec1bdbd94b3a8bfcf51b35d0198a
I20260620 06:05:58.968956 21005 raft_consensus.cc:2454] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1bffdd3a50d542d9b600da2547811617 in term 1.
I20260620 06:05:58.969084 21076 raft_consensus.cc:2726] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260620 06:05:58.969309 20673 leader_election.cc:302] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1bffdd3a50d542d9b600da2547811617, 9cc2ec1bdbd94b3a8bfcf51b35d0198a; no voters: ce3f09c6f4e344368d20da4080ed32bd
I20260620 06:05:58.969482 21074 raft_consensus.cc:2781] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:05:58.969720 21074 raft_consensus.cc:686] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [term 1 LEADER]: Becoming Leader. State: Replica: 1bffdd3a50d542d9b600da2547811617, State: Running, Role: LEADER
I20260620 06:05:58.970117 21074 consensus_queue.cc:237] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } }
I20260620 06:05:58.973436 20590 catalog_manager.cc:5495] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1bffdd3a50d542d9b600da2547811617 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "1bffdd3a50d542d9b600da2547811617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1bffdd3a50d542d9b600da2547811617" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 38439 } health_report { overall_health: HEALTHY } } }
I20260620 06:05:59.050072 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.046472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.3.115.1:38439)
0620 06:05:59.046624 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:59.046628 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:59.046645 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:05:59.046745 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:59.046747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:59.046801 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:59.046922 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:05:59.047163 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.047672 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.048532 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.048819 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.048943 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.049257 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:05:59.049696 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:05:59.049829 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:59.049865 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 06:05:59.049930 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:05:59.056354 21005 raft_consensus.cc:1261] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 1 FOLLOWER]: Refusing update from remote peer 1bffdd3a50d542d9b600da2547811617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:05:59.056354 20871 raft_consensus.cc:1261] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 1 FOLLOWER]: Refusing update from remote peer 1bffdd3a50d542d9b600da2547811617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:05:59.057003 21082 consensus_queue.cc:1035] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce3f09c6f4e344368d20da4080ed32bd" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 35575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:05:59.057193 21074 consensus_queue.cc:1035] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9cc2ec1bdbd94b3a8bfcf51b35d0198a" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 43859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:05:59.066396 21086 mvcc.cc:205] Tried to move back new op lower bound from 7298808049887203328 to 7298808049544294400. Current Snapshot: MvccSnapshot[applied={T|T < 7298808049887203328}]
I20260620 06:05:59.076247 20589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:05:59.076419 20589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:05:59.079071 20589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5c3a2d7d8cff4cb6bfefa56d62f14b93: Sending DeleteTablet for 3 replicas of tablet e4b310ec54224552ac1a2ff3d93b1b3d
I20260620 06:05:59.080025 20717 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b310ec54224552ac1a2ff3d93b1b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:05:59 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:38372
I20260620 06:05:59.080036 20985 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b310ec54224552ac1a2ff3d93b1b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:05:59 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:54108
I20260620 06:05:59.080603 20851 tablet_service.cc:1514] Processing DeleteTablet for tablet e4b310ec54224552ac1a2ff3d93b1b3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:05:59 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:51808
I20260620 06:05:59.080813 21095 tablet_replica.cc:330] T e4b310ec54224552ac1a2ff3d93b1b3d P 1bffdd3a50d542d9b600da2547811617: stopping tablet replica
I20260620 06:05:59.081072 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20654
I20260620 06:05:59.081122 21097 tablet_replica.cc:330] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: stopping tablet replica
I20260620 06:05:59.081487 21097 raft_consensus.cc:2229] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:05:59.081948 21097 raft_consensus.cc:2258] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:05:59.082861 21096 tablet_replica.cc:330] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: stopping tablet replica
I20260620 06:05:59.083175 21096 raft_consensus.cc:2229] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:05:59.083500 21096 raft_consensus.cc:2258] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:05:59.083772 21097 ts_tablet_manager.cc:1905] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:05:59.084826 21096 ts_tablet_manager.cc:1905] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:05:59.087280 21096 ts_tablet_manager.cc:1918] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:05:59.087400 21096 log.cc:1196] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/e4b310ec54224552ac1a2ff3d93b1b3d
I20260620 06:05:59.087689 21096 ts_tablet_manager.cc:1939] T e4b310ec54224552ac1a2ff3d93b1b3d P 9cc2ec1bdbd94b3a8bfcf51b35d0198a: Deleting consensus metadata
I20260620 06:05:59.087863 21097 ts_tablet_manager.cc:1918] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:05:59.087970 21097 log.cc:1196] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/e4b310ec54224552ac1a2ff3d93b1b3d
I20260620 06:05:59.088325 21097 ts_tablet_manager.cc:1939] T e4b310ec54224552ac1a2ff3d93b1b3d P ce3f09c6f4e344368d20da4080ed32bd: Deleting consensus metadata
I20260620 06:05:59.089143 20579 catalog_manager.cc:4841] TS 9cc2ec1bdbd94b3a8bfcf51b35d0198a (127.3.115.3:43859): tablet e4b310ec54224552ac1a2ff3d93b1b3d (table test-table [id=3e1f382242064aa6a1d1607104e522d4]) successfully deleted
I20260620 06:05:59.089643 20579 catalog_manager.cc:4841] TS ce3f09c6f4e344368d20da4080ed32bd (127.3.115.2:35575): tablet e4b310ec54224552ac1a2ff3d93b1b3d (table test-table [id=3e1f382242064aa6a1d1607104e522d4]) successfully deleted
W20260620 06:05:59.089804 20576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:38439 (error 108)
I20260620 06:05:59.090593 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20788
I20260620 06:05:59.091425 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.090658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:38439 (local address 127.0.0.1:38386)
0620 06:05:59.091223 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:59.091317 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:38439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453}
W20260620 06:05:59.091717 20576 catalog_manager.cc:4584] TS 1bffdd3a50d542d9b600da2547811617 (127.3.115.1:38439): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e4b310ec54224552ac1a2ff3d93b1b3d: Network error: Client connection negotiation failed: client connection to 127.3.115.1:38439: connect: Connection refused (error 111)
I20260620 06:05:59.097163 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20922
I20260620 06:05:59.102782 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 20560
2026-06-20T06:05:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2500 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T06:05:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:05:59Z Disabled control of system clock
I20260620 06:05:59.133585 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43403
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43403
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:05:59.246232 21102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:59.246541 21102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:59.246580 21102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:59.250662 21102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:05:59.250730 21102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:59.250746 21102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:59.250763 21102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:05:59.250780 21102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:05:59.255749 21102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:59.256920 21102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:59.258311 21102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:59.267383 21110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.267383 21108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:59.267493 21102 server_base.cc:1034] running on GCE node
W20260620 06:05:59.267433 21107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:59.268275 21102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:59.269502 21102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:59.270684 21102 hybrid_clock.cc:648] HybridClock initialized: now 1781935559270658 us; error 43 us; skew 500 ppm
I20260620 06:05:59.272857 21102 webserver.cc:460] Webserver started at http://127.3.115.62:36897/ using document root <none> and password file <none>
I20260620 06:05:59.273447 21102 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:59.273512 21102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:59.273682 21102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:59.275789 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "7f2c7c366c03406dab9f43279a703723"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "870f4102598a5a013661b22e2ef22c6b"
server_key_iv: "2179e0ac0879294c7db5c6e5f3b97e85"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.276300 21102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "7f2c7c366c03406dab9f43279a703723"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "870f4102598a5a013661b22e2ef22c6b"
server_key_iv: "2179e0ac0879294c7db5c6e5f3b97e85"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.280076 21102 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260620 06:05:59.283766 21116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.285554 21102 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260620 06:05:59.285689 21102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "7f2c7c366c03406dab9f43279a703723"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "870f4102598a5a013661b22e2ef22c6b"
server_key_iv: "2179e0ac0879294c7db5c6e5f3b97e85"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.285820 21102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:59.299708 21102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:59.300437 21102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:59.300596 21102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:59.307888 21102 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:43403
I20260620 06:05:59.307880 21168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:43403 every 8 connection(s)
I20260620 06:05:59.309089 21102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:05:59.309491 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21102
I20260620 06:05:59.309662 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:05:59.310063 3532 external_mini_cluster.cc:1428] Setting key ad256b2873a0702b1c4b980404d80641
I20260620 06:05:59.313081 21169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:05:59.318743 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723: Bootstrap starting.
I20260620 06:05:59.319720 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.311618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53642 (local address 127.3.115.62:43403)
0620 06:05:59.312315 (+ 697us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:59.312331 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:59.312377 (+ 46us) server_negotiation.cc:407] Connection header received
0620 06:05:59.313093 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:59.313113 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:59.313578 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:59.313944 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:59.314397 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.315370 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.316304 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.316851 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.317031 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.317928 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:59.317933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:59.317946 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:05:59.318361 (+ 415us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:59.318418 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:05:59.318438 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:59.318544 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:59.318677 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 06:05:59.318974 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":153,"threads_started":1}
I20260620 06:05:59.321319 21169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723: Neither blocks nor log segments found. Creating new log.
I20260620 06:05:59.322245 21169 log.cc:824] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723: Log is configured to *not* fsync() on all Append() calls
I20260620 06:05:59.324477 21169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723: No bootstrap required, opened a new log
I20260620 06:05:59.328239 21169 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c7c366c03406dab9f43279a703723" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43403 } }
I20260620 06:05:59.328486 21169 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:05:59.328531 21169 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f2c7c366c03406dab9f43279a703723, State: Initialized, Role: FOLLOWER
I20260620 06:05:59.329042 21169 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c7c366c03406dab9f43279a703723" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43403 } }
I20260620 06:05:59.329228 21169 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:05:59.329304 21169 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:05:59.329509 21169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:05:59.330544 21169 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c7c366c03406dab9f43279a703723" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43403 } }
I20260620 06:05:59.330886 21169 leader_election.cc:302] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f2c7c366c03406dab9f43279a703723; no voters:
I20260620 06:05:59.331221 21169 leader_election.cc:288] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:05:59.331380 21174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:05:59.331624 21174 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [term 1 LEADER]: Becoming Leader. State: Replica: 7f2c7c366c03406dab9f43279a703723, State: Running, Role: LEADER
I20260620 06:05:59.331914 21174 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c7c366c03406dab9f43279a703723" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43403 } }
I20260620 06:05:59.332505 21169 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:05:59.333706 21175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f2c7c366c03406dab9f43279a703723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c7c366c03406dab9f43279a703723" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43403 } } }
I20260620 06:05:59.333837 21175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:59.334111 21176 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f2c7c366c03406dab9f43279a703723. Latest consensus state: current_term: 1 leader_uuid: "7f2c7c366c03406dab9f43279a703723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f2c7c366c03406dab9f43279a703723" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43403 } } }
I20260620 06:05:59.334220 21176 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723 [sys.catalog]: This master's current role is: LEADER
I20260620 06:05:59.334189 21183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:05:59.337230 21183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:05:59.343061 21183 catalog_manager.cc:1347] Generated new cluster ID: 526124d237374faabc5f185e282ac54a
I20260620 06:05:59.343142 21183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:05:59.349924 21183 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:05:59.351189 21183 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:05:59.357196 21183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723: Generated new TSK 0
I20260620 06:05:59.358068 21183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:05:59.361775 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43403
--builtin_ntp_servers=127.3.115.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:05:59.477871 21193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:59.478181 21193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:59.478260 21193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:59.482509 21193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:59.482604 21193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:59.482695 21193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:05:59.487826 21193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21193
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:59.489100 21193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:59.490594 21193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:59.500150 21201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.500182 21199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.500145 21198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:59.500453 21193 server_base.cc:1034] running on GCE node
I20260620 06:05:59.500924 21193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:59.501611 21193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:59.502794 21193 hybrid_clock.cc:648] HybridClock initialized: now 1781935559502771 us; error 32 us; skew 500 ppm
I20260620 06:05:59.509155 21193 webserver.cc:460] Webserver started at http://127.3.115.1:38359/ using document root <none> and password file <none>
I20260620 06:05:59.510062 21193 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:59.510175 21193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:59.510471 21193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:59.513844 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "38cbcc00bf9e4852a3252a3596f309f1"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "967522ce55190e8ad322f7543b7256f7"
server_key_iv: "2e5aaa89618e9bc2887dfd0cabea555f"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.514586 21193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "38cbcc00bf9e4852a3252a3596f309f1"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "967522ce55190e8ad322f7543b7256f7"
server_key_iv: "2e5aaa89618e9bc2887dfd0cabea555f"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.519937 21193 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260620 06:05:59.523590 21207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.525252 21193 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260620 06:05:59.525393 21193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "38cbcc00bf9e4852a3252a3596f309f1"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "967522ce55190e8ad322f7543b7256f7"
server_key_iv: "2e5aaa89618e9bc2887dfd0cabea555f"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.525542 21193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:59.544497 21193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:59.545367 21193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:59.545583 21193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:59.546241 21193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:59.547310 21193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:59.547376 21193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.547441 21193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:59.547474 21193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.558710 21193 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:42327
I20260620 06:05:59.558753 21320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:42327 every 8 connection(s)
I20260620 06:05:59.559870 21193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:05:59.568724 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21193
I20260620 06:05:59.568850 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:05:59.569126 3532 external_mini_cluster.cc:1428] Setting key bc5f08e47f3324a0f908dd7e11587cdd
I20260620 06:05:59.571429 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43403
--builtin_ntp_servers=127.3.115.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:05:59.571615 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.563572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:32981 (local address 127.3.115.62:43403)
0620 06:05:59.563699 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:59.563703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:59.563721 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:05:59.565126 (+ 1405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:59.565130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:59.565190 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:59.565298 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:59.566242 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.566984 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.568170 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.568399 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.568489 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.570974 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:59.570978 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:59.570981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:05:59.570994 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:59.571019 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:05:59.571029 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:59.571121 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:59.571348 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 06:05:59.571443 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:05:59.572512 21323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.562456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43403 (local address 127.3.115.1:32981)
0620 06:05:59.563255 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:59.563310 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:59.564907 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:59.565459 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:59.565469 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:59.565583 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:59.566103 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:59.566115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.567103 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.567105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:59.568042 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:59.568051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.569641 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.569648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:59.569832 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.570621 (+ 789us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:05:59.570623 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:59.570684 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:05:59.570775 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:59.571143 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:59.571153 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:59.571163 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 06:05:59.571324 (+ 161us) client_negotiation.cc:240] Negotiation successful
0620 06:05:59.571633 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":142,"threads_started":1}
I20260620 06:05:59.573738 21321 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43403
I20260620 06:05:59.573985 21321 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:59.574702 21321 heartbeater.cc:511] Master 127.3.115.62:43403 requested a full tablet report, sending...
I20260620 06:05:59.576457 21130 ts_manager.cc:194] Registered new tserver with Master: 38cbcc00bf9e4852a3252a3596f309f1 (127.3.115.1:42327)
I20260620 06:05:59.578256 21130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.1:32981
W20260620 06:05:59.688345 21324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:59.688630 21324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:59.688738 21324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:59.694096 21324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:59.694161 21324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:59.694221 21324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:05:59.699178 21324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:59.700259 21324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:59.702064 21324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:59.712011 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.712044 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.712044 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:59.712293 21324 server_base.cc:1034] running on GCE node
I20260620 06:05:59.712877 21324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:59.713531 21324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:59.714718 21324 hybrid_clock.cc:648] HybridClock initialized: now 1781935559714708 us; error 47 us; skew 500 ppm
I20260620 06:05:59.716643 21324 webserver.cc:460] Webserver started at http://127.3.115.2:43077/ using document root <none> and password file <none>
I20260620 06:05:59.717279 21324 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:59.717368 21324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:59.717607 21324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:59.719621 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "3d09a8698fa8470a8857be16cb4a278e"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "14bd659ba3c8ce99a350802a11615080"
server_key_iv: "8d18dbd9069dc5dd1dcd831b9aad22d6"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.720211 21324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "3d09a8698fa8470a8857be16cb4a278e"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "14bd659ba3c8ce99a350802a11615080"
server_key_iv: "8d18dbd9069dc5dd1dcd831b9aad22d6"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.723769 21324 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:05:59.726250 21338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.727490 21324 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:05:59.727628 21324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "3d09a8698fa8470a8857be16cb4a278e"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "14bd659ba3c8ce99a350802a11615080"
server_key_iv: "8d18dbd9069dc5dd1dcd831b9aad22d6"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.727762 21324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:59.757529 21324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:59.758369 21324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:59.758575 21324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:59.759205 21324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:59.760236 21324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:59.760308 21324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.760385 21324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:59.760426 21324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.770694 21324 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:46471
I20260620 06:05:59.770721 21451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:46471 every 8 connection(s)
I20260620 06:05:59.771680 21324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:05:59.778681 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21324
I20260620 06:05:59.778820 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:05:59.779153 3532 external_mini_cluster.cc:1428] Setting key 3e974fb189e2e4b3897aaa003b4b7aaa
I20260620 06:05:59.781247 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43403
--builtin_ntp_servers=127.3.115.20:33285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:05:59.782275 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.773658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:35415 (local address 127.3.115.62:43403)
0620 06:05:59.773832 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:59.773838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:59.774973 (+ 1135us) server_negotiation.cc:407] Connection header received
0620 06:05:59.776476 (+ 1503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:59.776479 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:59.776538 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:59.776621 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:59.777597 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.778143 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.779846 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.780109 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.780267 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.781712 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:59.781717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:59.781720 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:05:59.781739 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:59.781763 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:05:59.781773 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:59.781855 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:59.782037 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:05:59.782114 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:05:59.783293 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.774124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43403 (local address 127.3.115.2:35415)
0620 06:05:59.774806 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:59.774852 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:59.776223 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:59.776817 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:59.776824 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:59.776911 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:59.777455 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:59.777468 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.778268 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.778271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:59.779675 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:59.779685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.780317 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.780321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:59.780571 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.781349 (+ 778us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:05:59.781351 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:59.781432 (+ 81us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:05:59.781532 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:59.781870 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:59.781882 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:59.781888 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:05:59.782036 (+ 148us) client_negotiation.cc:240] Negotiation successful
0620 06:05:59.782360 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":127,"threads_started":1}
I20260620 06:05:59.784543 21452 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43403
I20260620 06:05:59.784837 21452 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:59.785432 21452 heartbeater.cc:511] Master 127.3.115.62:43403 requested a full tablet report, sending...
I20260620 06:05:59.786592 21130 ts_manager.cc:194] Registered new tserver with Master: 3d09a8698fa8470a8857be16cb4a278e (127.3.115.2:46471)
I20260620 06:05:59.787498 21130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.2:35415
W20260620 06:05:59.895399 21455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:05:59.895684 21455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:05:59.895723 21455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:05:59.899855 21455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:05:59.899924 21455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:05:59.899982 21455 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:05:59.904778 21455 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:43403
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:05:59.905894 21455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:05:59.907408 21455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:05:59.917620 21460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.917804 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:05:59.917788 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:05:59.917883 21455 server_base.cc:1034] running on GCE node
I20260620 06:05:59.918445 21455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:05:59.919114 21455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:05:59.920333 21455 hybrid_clock.cc:648] HybridClock initialized: now 1781935559920296 us; error 50 us; skew 500 ppm
I20260620 06:05:59.922469 21455 webserver.cc:460] Webserver started at http://127.3.115.3:44819/ using document root <none> and password file <none>
I20260620 06:05:59.923084 21455 fs_manager.cc:359] Metadata directory not provided
I20260620 06:05:59.923151 21455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:05:59.923375 21455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:05:59.925611 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "0f69c33171fe455aac8160b6d5575a35"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "71b78295ec728ad3c0a4c1d68c2fb5b7"
server_key_iv: "0d14888307188feda1dc97f4571ca38d"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.926189 21455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "0f69c33171fe455aac8160b6d5575a35"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "71b78295ec728ad3c0a4c1d68c2fb5b7"
server_key_iv: "0d14888307188feda1dc97f4571ca38d"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.930153 21455 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:05:59.933112 21469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.934664 21455 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:05:59.934798 21455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "0f69c33171fe455aac8160b6d5575a35"
format_stamp: "Formatted at 2026-06-20 06:05:59 on dist-test-slave-4h14"
server_key: "71b78295ec728ad3c0a4c1d68c2fb5b7"
server_key_iv: "0d14888307188feda1dc97f4571ca38d"
server_key_version: "encryptionkey@0"
I20260620 06:05:59.934927 21455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:05:59.951107 21455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:05:59.951880 21455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:05:59.952085 21455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:05:59.952803 21455 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:05:59.953802 21455 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:05:59.953881 21455 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.953954 21455 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:05:59.953994 21455 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:05:59.963474 21455 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:44181
I20260620 06:05:59.963547 21582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:44181 every 8 connection(s)
I20260620 06:05:59.964489 21455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:05:59.967929 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21455
I20260620 06:05:59.968048 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:05:59.968319 3532 external_mini_cluster.cc:1428] Setting key 5b9da8bfc658a0f9ea8eebfca6059f9d
I20260620 06:05:59.974357 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.966489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:50623 (local address 127.3.115.62:43403)
0620 06:05:59.966637 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:59.966641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:59.967693 (+ 1052us) server_negotiation.cc:407] Connection header received
0620 06:05:59.969446 (+ 1753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:59.969449 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:59.969505 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:59.969582 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:59.970505 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.971037 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.972218 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.972444 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.972556 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.973846 (+ 1290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:59.973849 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:59.973851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:05:59.973864 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:59.973884 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:05:59.973892 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:59.973979 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:59.974143 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 06:05:59.974211 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:05:59.975428 21585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.966845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43403 (local address 127.3.115.3:50623)
0620 06:05:59.967528 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0620 06:05:59.967571 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:05:59.969222 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:05:59.969770 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:05:59.969778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:05:59.969867 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:05:59.970364 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:59.970377 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.971165 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.971169 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:59.972092 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:05:59.972101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.972593 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.972597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:05:59.972800 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.973507 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:05:59.973509 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:05:59.973564 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:05:59.973631 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:05:59.973984 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:05:59.973995 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:05:59.974003 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:05:59.974143 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:05:59.974458 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":108,"threads_started":1}
I20260620 06:05:59.976594 21583 heartbeater.cc:347] Connected to a master server at 127.3.115.62:43403
I20260620 06:05:59.976895 21583 heartbeater.cc:464] Registering TS with master...
I20260620 06:05:59.977448 21583 heartbeater.cc:511] Master 127.3.115.62:43403 requested a full tablet report, sending...
I20260620 06:05:59.978581 21130 ts_manager.cc:194] Registered new tserver with Master: 0f69c33171fe455aac8160b6d5575a35 (127.3.115.3:44181)
I20260620 06:05:59.979156 21130 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.3:50623
I20260620 06:05:59.982309 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:05:59.988207 21172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:05:59.984268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53648 (local address 127.3.115.62:43403)
0620 06:05:59.984414 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:05:59.984418 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:05:59.984608 (+ 190us) server_negotiation.cc:407] Connection header received
0620 06:05:59.984804 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:05:59.984807 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:05:59.984849 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:05:59.984917 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:05:59.985166 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.985684 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.986616 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:05:59.986869 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:05:59.986968 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:05:59.987765 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:05:59.987768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:05:59.987770 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:05:59.987783 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:05:59.987801 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:05:59.987809 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:05:59.987870 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 06:05:59.987963 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 06:05:59.988013 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:05:59.991163 21130 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53648:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:05:59.995079 21130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:00.010416 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.004123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:42327 (local address 127.0.0.1:49562)
0620 06:06:00.005024 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:00.005071 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:00.005238 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:00.006079 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:00.006082 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:00.006096 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:00.006190 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.006194 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.007349 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.007353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.008814 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.008826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.009770 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.009774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.009899 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.009940 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 06:06:00.010084 (+ 144us) client_negotiation.cc:240] Negotiation successful
0620 06:06:00.010221 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":824,"mutex_wait_us":152,"spinlock_wait_cycles":7680,"thread_start_us":116,"threads_started":1}
I20260620 06:06:00.010457 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.004116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:46471 (local address 127.0.0.1:36532)
0620 06:06:00.005011 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:00.005066 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:00.005238 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:00.006005 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:00.006011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:00.006037 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:00.006191 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.006198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.007609 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.007612 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.008878 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.008887 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.010087 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.010092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.010192 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.010212 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:06:00.010284 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 06:06:00.010343 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":70,"spinlock_wait_cycles":5760,"thread_start_us":69,"threads_started":1}
I20260620 06:06:00.010746 21599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.004943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:44181 (local address 127.0.0.1:52748)
0620 06:06:00.005904 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:00.005919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:00.006003 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:00.006501 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:00.006504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:00.006522 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:00.006594 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.006599 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.007905 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.007908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.009092 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.009103 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.010353 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.010357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.010441 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.010463 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:06:00.010537 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 06:06:00.010618 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":590,"thread_start_us":88,"threads_started":1}
I20260620 06:06:00.010871 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.004391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49562 (local address 127.3.115.1:42327)
0620 06:06:00.004862 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:00.004881 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:00.005406 (+ 525us) server_negotiation.cc:407] Connection header received
0620 06:06:00.005496 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:00.005503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:00.005742 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:00.005907 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:00.006304 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.007228 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.008944 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.009633 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.009775 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.010005 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:00.010565 (+ 560us) server_negotiation.cc:299] Negotiation successful
0620 06:06:00.010672 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":110,"threads_started":1}
I20260620 06:06:00.011225 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.004227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.3.115.2:46471)
0620 06:06:00.004956 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:00.004968 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:00.005407 (+ 439us) server_negotiation.cc:407] Connection header received
0620 06:06:00.005493 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:00.005499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:00.005743 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:00.005908 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:00.006330 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.007495 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.009014 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.009790 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.009916 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.010118 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:00.010338 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 06:06:00.010457 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":129,"threads_started":1}
I20260620 06:06:00.011886 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.005178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.3.115.3:44181)
0620 06:06:00.005908 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:00.005920 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:00.006070 (+ 150us) server_negotiation.cc:407] Connection header received
0620 06:06:00.006146 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:00.006151 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:00.006360 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:00.006506 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:00.006720 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.007780 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.009235 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.010017 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.010114 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.010277 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:00.010923 (+ 646us) server_negotiation.cc:299] Negotiation successful
0620 06:06:00.011711 (+ 788us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":174,"threads_started":1}
I20260620 06:06:00.014202 21517 tablet_service.cc:1467] Processing CreateTablet for tablet 635e551447a8437fbe2d8d0a2573fda8 (DEFAULT_TABLE table=test-table [id=a6a7d774cd914ff4bec33156e802e221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:00.014210 21255 tablet_service.cc:1467] Processing CreateTablet for tablet 635e551447a8437fbe2d8d0a2573fda8 (DEFAULT_TABLE table=test-table [id=a6a7d774cd914ff4bec33156e802e221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:00.014209 21386 tablet_service.cc:1467] Processing CreateTablet for tablet 635e551447a8437fbe2d8d0a2573fda8 (DEFAULT_TABLE table=test-table [id=a6a7d774cd914ff4bec33156e802e221]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:00.015271 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635e551447a8437fbe2d8d0a2573fda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:00.015276 21386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635e551447a8437fbe2d8d0a2573fda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:00.015276 21255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 635e551447a8437fbe2d8d0a2573fda8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:00.022603 21600 tablet_bootstrap.cc:492] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Bootstrap starting.
I20260620 06:06:00.023545 21602 tablet_bootstrap.cc:492] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Bootstrap starting.
I20260620 06:06:00.024778 21600 tablet_bootstrap.cc:654] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:00.025013 21601 tablet_bootstrap.cc:492] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: Bootstrap starting.
I20260620 06:06:00.025590 21602 tablet_bootstrap.cc:654] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:00.025624 21600 log.cc:824] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:00.026434 21602 log.cc:824] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:00.027171 21601 tablet_bootstrap.cc:654] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:00.027884 21600 tablet_bootstrap.cc:492] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: No bootstrap required, opened a new log
I20260620 06:06:00.028052 21601 log.cc:824] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:00.028098 21600 ts_tablet_manager.cc:1397] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:00.029995 21602 tablet_bootstrap.cc:492] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: No bootstrap required, opened a new log
I20260620 06:06:00.030218 21602 ts_tablet_manager.cc:1397] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260620 06:06:00.030354 21601 tablet_bootstrap.cc:492] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: No bootstrap required, opened a new log
I20260620 06:06:00.030537 21601 ts_tablet_manager.cc:1397] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:00.032399 21600 raft_consensus.cc:348] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.032711 21600 raft_consensus.cc:374] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:00.032783 21600 raft_consensus.cc:729] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f69c33171fe455aac8160b6d5575a35, State: Initialized, Role: FOLLOWER
I20260620 06:06:00.033299 21600 consensus_queue.cc:260] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.034013 21583 heartbeater.cc:503] Master 127.3.115.62:43403 was elected leader, sending a full tablet report...
I20260620 06:06:00.034435 21600 ts_tablet_manager.cc:1428] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260620 06:06:00.034824 21601 raft_consensus.cc:348] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.035315 21601 raft_consensus.cc:374] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:00.035399 21601 raft_consensus.cc:729] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38cbcc00bf9e4852a3252a3596f309f1, State: Initialized, Role: FOLLOWER
I20260620 06:06:00.035858 21601 consensus_queue.cc:260] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.035848 21602 raft_consensus.cc:348] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.036087 21602 raft_consensus.cc:374] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:00.036144 21602 raft_consensus.cc:729] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d09a8698fa8470a8857be16cb4a278e, State: Initialized, Role: FOLLOWER
I20260620 06:06:00.036541 21602 consensus_queue.cc:260] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.036968 21601 ts_tablet_manager.cc:1428] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260620 06:06:00.037259 21452 heartbeater.cc:503] Master 127.3.115.62:43403 was elected leader, sending a full tablet report...
I20260620 06:06:00.037295 21321 heartbeater.cc:503] Master 127.3.115.62:43403 was elected leader, sending a full tablet report...
I20260620 06:06:00.037504 21602 ts_tablet_manager.cc:1428] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260620 06:06:00.061209 21322 tablet.cc:2369] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:00.149214 21608 raft_consensus.cc:482] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:00.149446 21608 raft_consensus.cc:504] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.150689 21608 leader_election.cc:288] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38cbcc00bf9e4852a3252a3596f309f1 (127.3.115.1:42327), 0f69c33171fe455aac8160b6d5575a35 (127.3.115.3:44181)
I20260620 06:06:00.155799 21597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.151112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:48075 (local address 127.3.115.1:42327)
0620 06:06:00.151238 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:00.151243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:00.151260 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:06:00.151434 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:00.151437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:00.151495 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:00.151576 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:00.151965 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.152804 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.154441 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.155092 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.155208 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.155275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:00.155616 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 06:06:00.155698 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:06:00.155787 21454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.151012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:42327 (local address 127.3.115.2:48075)
0620 06:06:00.151126 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:00.151142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:00.151247 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:00.151657 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:00.151662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:00.151685 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:00.151820 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.151831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.152951 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.152954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.154205 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.154217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.155242 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.155246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.155392 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.155430 (+ 38us) client_negotiation.cc:769] Sending connection context
0620 06:06:00.155518 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 06:06:00.155639 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":14}
I20260620 06:06:00.155804 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.151226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:58829 (local address 127.3.115.3:44181)
0620 06:06:00.151374 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:00.151378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:00.151924 (+ 546us) server_negotiation.cc:407] Connection header received
0620 06:06:00.151991 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:00.151994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:00.152045 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:00.152135 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:00.152365 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.153063 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.154465 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.155092 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.155208 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.155275 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:00.155573 (+ 298us) server_negotiation.cc:299] Negotiation successful
0620 06:06:00.155660 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 06:06:00.156343 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.151012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:44181 (local address 127.3.115.2:58829)
0620 06:06:00.151770 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:00.151787 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:00.151912 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:00.152147 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:00.152149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:00.152161 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:00.152254 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.152258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.153174 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.153177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.154335 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:00.154345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.155242 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.155246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:00.155397 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.155452 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 06:06:00.155518 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:06:00.156167 (+ 649us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"mutex_wait_us":1009,"thread_start_us":119,"threads_started":1}
I20260620 06:06:00.156621 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635e551447a8437fbe2d8d0a2573fda8" candidate_uuid: "3d09a8698fa8470a8857be16cb4a278e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cbcc00bf9e4852a3252a3596f309f1" is_pre_election: true
I20260620 06:06:00.157027 21275 raft_consensus.cc:2454] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d09a8698fa8470a8857be16cb4a278e in term 0.
I20260620 06:06:00.157042 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635e551447a8437fbe2d8d0a2573fda8" candidate_uuid: "3d09a8698fa8470a8857be16cb4a278e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f69c33171fe455aac8160b6d5575a35" is_pre_election: true
I20260620 06:06:00.157368 21537 raft_consensus.cc:2454] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d09a8698fa8470a8857be16cb4a278e in term 0.
I20260620 06:06:00.157567 21341 leader_election.cc:302] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38cbcc00bf9e4852a3252a3596f309f1, 3d09a8698fa8470a8857be16cb4a278e; no voters:
I20260620 06:06:00.157903 21608 raft_consensus.cc:2781] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:00.157984 21608 raft_consensus.cc:482] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:00.158016 21608 raft_consensus.cc:3037] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:00.159152 21608 raft_consensus.cc:504] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.159541 21608 leader_election.cc:288] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [CANDIDATE]: Term 1 election: Requested vote from peers 38cbcc00bf9e4852a3252a3596f309f1 (127.3.115.1:42327), 0f69c33171fe455aac8160b6d5575a35 (127.3.115.3:44181)
I20260620 06:06:00.159894 21537 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635e551447a8437fbe2d8d0a2573fda8" candidate_uuid: "3d09a8698fa8470a8857be16cb4a278e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f69c33171fe455aac8160b6d5575a35"
I20260620 06:06:00.159950 21275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "635e551447a8437fbe2d8d0a2573fda8" candidate_uuid: "3d09a8698fa8470a8857be16cb4a278e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38cbcc00bf9e4852a3252a3596f309f1"
I20260620 06:06:00.160048 21537 raft_consensus.cc:3037] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:00.160077 21275 raft_consensus.cc:3037] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:00.161238 21537 raft_consensus.cc:2454] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d09a8698fa8470a8857be16cb4a278e in term 1.
I20260620 06:06:00.161247 21275 raft_consensus.cc:2454] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d09a8698fa8470a8857be16cb4a278e in term 1.
I20260620 06:06:00.161590 21342 leader_election.cc:302] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f69c33171fe455aac8160b6d5575a35, 3d09a8698fa8470a8857be16cb4a278e; no voters:
I20260620 06:06:00.161767 21608 raft_consensus.cc:2781] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:00.162020 21608 raft_consensus.cc:686] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 1 LEADER]: Becoming Leader. State: Replica: 3d09a8698fa8470a8857be16cb4a278e, State: Running, Role: LEADER
I20260620 06:06:00.162421 21608 consensus_queue.cc:237] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } }
I20260620 06:06:00.165823 21133 catalog_manager.cc:5495] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d09a8698fa8470a8857be16cb4a278e (127.3.115.2). New cstate: current_term: 1 leader_uuid: "3d09a8698fa8470a8857be16cb4a278e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d09a8698fa8470a8857be16cb4a278e" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 } health_report { overall_health: UNKNOWN } } }
I20260620 06:06:00.202036 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.198596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.3.115.2:46471)
0620 06:06:00.198734 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:00.198738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:00.198753 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:00.198829 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:00.198832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:00.198888 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:00.198986 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:00.199188 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.199725 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.200538 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:00.200771 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:00.200856 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:00.201126 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:00.201589 (+ 463us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:00.201681 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:00.201786 (+ 105us) server_negotiation.cc:299] Negotiation successful
0620 06:06:00.201831 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260620 06:06:00.207937 21537 raft_consensus.cc:1261] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 1 FOLLOWER]: Refusing update from remote peer 3d09a8698fa8470a8857be16cb4a278e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:00.207938 21275 raft_consensus.cc:1261] T 635e551447a8437fbe2d8d0a2573fda8 P 38cbcc00bf9e4852a3252a3596f309f1 [term 1 FOLLOWER]: Refusing update from remote peer 3d09a8698fa8470a8857be16cb4a278e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:00.208727 21610 consensus_queue.cc:1035] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38cbcc00bf9e4852a3252a3596f309f1" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:00.208945 21608 consensus_queue.cc:1035] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f69c33171fe455aac8160b6d5575a35" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 44181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:06:00.215931 21584 tablet.cc:2369] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:00.216466 21613 mvcc.cc:205] Tried to move back new op lower bound from 7298808054604533760 to 7298808054428327936. Current Snapshot: MvccSnapshot[applied={T|T < 7298808054604533760}]
I20260620 06:06:00.218008 21614 mvcc.cc:205] Tried to move back new op lower bound from 7298808054604533760 to 7298808054428327936. Current Snapshot: MvccSnapshot[applied={T|T < 7298808054604533760}]
I20260620 06:06:00.218282 21615 mvcc.cc:205] Tried to move back new op lower bound from 7298808054604533760 to 7298808054428327936. Current Snapshot: MvccSnapshot[applied={T|T < 7298808054604533760}]
I20260620 06:06:00.225941 21133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:00.226126 21133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53648:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:00.228775 21133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f2c7c366c03406dab9f43279a703723: Sending DeleteTablet for 3 replicas of tablet 635e551447a8437fbe2d8d0a2573fda8
I20260620 06:06:00.229681 21255 tablet_service.cc:1514] Processing DeleteTablet for tablet 635e551447a8437fbe2d8d0a2573fda8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:00 UTC) from {username='slave'} at 127.0.0.1:49562
I20260620 06:06:00.229746 21386 tablet_service.cc:1514] Processing DeleteTablet for tablet 635e551447a8437fbe2d8d0a2573fda8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:00 UTC) from {username='slave'} at 127.0.0.1:36532
I20260620 06:06:00.230263 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21193
I20260620 06:06:00.230566 21623 tablet_replica.cc:330] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: stopping tablet replica
I20260620 06:06:00.230911 21517 tablet_service.cc:1514] Processing DeleteTablet for tablet 635e551447a8437fbe2d8d0a2573fda8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:00 UTC) from {username='slave'} at 127.0.0.1:52748
I20260620 06:06:00.231020 21623 raft_consensus.cc:2229] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:00.231480 21623 raft_consensus.cc:2258] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:00.231926 21624 tablet_replica.cc:330] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: stopping tablet replica
I20260620 06:06:00.232415 21624 raft_consensus.cc:2229] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:00.232839 21624 raft_consensus.cc:2258] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:00.234169 21623 ts_tablet_manager.cc:1905] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:00.235496 21624 ts_tablet_manager.cc:1905] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:00.236979 21623 ts_tablet_manager.cc:1918] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:00.237072 21623 log.cc:1196] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/635e551447a8437fbe2d8d0a2573fda8
I20260620 06:06:00.237371 21623 ts_tablet_manager.cc:1939] T 635e551447a8437fbe2d8d0a2573fda8 P 3d09a8698fa8470a8857be16cb4a278e: Deleting consensus metadata
I20260620 06:06:00.238600 21120 catalog_manager.cc:4841] TS 3d09a8698fa8470a8857be16cb4a278e (127.3.115.2:46471): tablet 635e551447a8437fbe2d8d0a2573fda8 (table test-table [id=a6a7d774cd914ff4bec33156e802e221]) successfully deleted
I20260620 06:06:00.238956 21624 ts_tablet_manager.cc:1918] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:00.239060 21624 log.cc:1196] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/635e551447a8437fbe2d8d0a2573fda8
I20260620 06:06:00.239377 21624 ts_tablet_manager.cc:1939] T 635e551447a8437fbe2d8d0a2573fda8 P 0f69c33171fe455aac8160b6d5575a35: Deleting consensus metadata
W20260620 06:06:00.239970 21119 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:42327 (error 108)
I20260620 06:06:00.240715 21120 catalog_manager.cc:4841] TS 0f69c33171fe455aac8160b6d5575a35 (127.3.115.3:44181): tablet 635e551447a8437fbe2d8d0a2573fda8 (table test-table [id=a6a7d774cd914ff4bec33156e802e221]) successfully deleted
I20260620 06:06:00.240864 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21324
I20260620 06:06:00.241160 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:00.240763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:42327 (local address 127.0.0.1:49568)
0620 06:06:00.240974 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:00.241057 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
W20260620 06:06:00.242660 21119 catalog_manager.cc:4584] TS 38cbcc00bf9e4852a3252a3596f309f1 (127.3.115.1:42327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 635e551447a8437fbe2d8d0a2573fda8: Network error: Client connection negotiation failed: client connection to 127.3.115.1:42327: connect: Connection refused (error 111)
I20260620 06:06:00.248695 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21455
I20260620 06:06:00.254482 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21102
2026-06-20T06:06:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1150 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:00 dist-test-slave-4h14 krb5kdc[21628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:00 dist-test-slave-4h14 krb5kdc[21628](info): set up 2 sockets
Jun 20 06:06:00 dist-test-slave-4h14 krb5kdc[21628](info): commencing operation
krb5kdc: starting...
W20260620 06:06:01.296536 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:01 dist-test-slave-4h14 krb5kdc[21628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:01.419795 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45007
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:45007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:01.533219 21641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:01.533526 21641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:01.533613 21641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:01.537778 21641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:01.537870 21641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:01.537911 21641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:01.537967 21641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:01.538007 21641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:01.543342 21641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:45007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:01.544478 21641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:01.545697 21641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:01.554729 21647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:01.554683 21649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:01.554767 21641 server_base.cc:1034] running on GCE node
W20260620 06:06:01.554705 21646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:01.555696 21641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:01.557029 21641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:01.558259 21641 hybrid_clock.cc:648] HybridClock initialized: now 1781935561558239 us; error 39 us; skew 500 ppm
Jun 20 06:06:01 dist-test-slave-4h14 krb5kdc[21628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935561, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:01.561255 21641 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:01.562232 21641 webserver.cc:460] Webserver started at http://127.3.115.62:39683/ using document root <none> and password file <none>
I20260620 06:06:01.562803 21641 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:01.562892 21641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:01.563122 21641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:01.565322 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "60bee7dfe01f43ad94bdc847ee03ba45"
format_stamp: "Formatted at 2026-06-20 06:06:01 on dist-test-slave-4h14"
server_key: "b07707583b4c8e0acf9820172da814c9"
server_key_iv: "1f4ca2adc32ebc42350999bb6c4f4e1b"
server_key_version: "encryptionkey@0"
I20260620 06:06:01.565932 21641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "60bee7dfe01f43ad94bdc847ee03ba45"
format_stamp: "Formatted at 2026-06-20 06:06:01 on dist-test-slave-4h14"
server_key: "b07707583b4c8e0acf9820172da814c9"
server_key_iv: "1f4ca2adc32ebc42350999bb6c4f4e1b"
server_key_version: "encryptionkey@0"
I20260620 06:06:01.569772 21641 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:06:01.572208 21656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:01.573560 21641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:01.573709 21641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "60bee7dfe01f43ad94bdc847ee03ba45"
format_stamp: "Formatted at 2026-06-20 06:06:01 on dist-test-slave-4h14"
server_key: "b07707583b4c8e0acf9820172da814c9"
server_key_iv: "1f4ca2adc32ebc42350999bb6c4f4e1b"
server_key_version: "encryptionkey@0"
I20260620 06:06:01.573848 21641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:01.607371 21641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:01.610517 21641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:01.610745 21641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:01.618026 21641 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:45007
I20260620 06:06:01.618042 21708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:45007 every 8 connection(s)
I20260620 06:06:01.619040 21641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:01.621974 21709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:01.626583 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21641
I20260620 06:06:01.626729 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:01.627041 3532 external_mini_cluster.cc:1428] Setting key 9a5d2d721166a420e5b20a3d07823ee3
I20260620 06:06:01.628096 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45: Bootstrap starting.
I20260620 06:06:01.630959 21709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:01.631860 21709 log.cc:824] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:01.634080 21709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45: No bootstrap required, opened a new log
Jun 20 06:06:01 dist-test-slave-4h14 krb5kdc[21628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:01.638644 21709 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45007 } }
I20260620 06:06:01.638985 21709 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:01.639061 21709 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60bee7dfe01f43ad94bdc847ee03ba45, State: Initialized, Role: FOLLOWER
I20260620 06:06:01.639633 21709 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45007 } }
I20260620 06:06:01.639807 21709 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:01.639875 21709 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:01.639981 21709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:01.641203 21709 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45007 } }
I20260620 06:06:01.641583 21709 leader_election.cc:302] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60bee7dfe01f43ad94bdc847ee03ba45; no voters:
I20260620 06:06:01.641942 21709 leader_election.cc:288] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:01.642103 21714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:01.642331 21714 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [term 1 LEADER]: Becoming Leader. State: Replica: 60bee7dfe01f43ad94bdc847ee03ba45, State: Running, Role: LEADER
I20260620 06:06:01.642699 21714 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45007 } }
I20260620 06:06:01.643255 21709 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:01.644016 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:01.628707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.3.115.62:45007)
0620 06:06:01.629144 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:01.629164 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:01.629212 (+ 48us) server_negotiation.cc:407] Connection header received
0620 06:06:01.630103 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:01.630129 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:01.630588 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:01.631006 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:01.631799 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:01.633170 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:01.634073 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:01.634799 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:01.634963 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:01.637982 (+ 3019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:01.637989 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:01.638010 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:01.638049 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:01.640323 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:01.640862 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:01.640870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:01.640876 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:01.641033 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:01.641371 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:01.641375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:01.641376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:01.641928 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:01.642136 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:01.642652 (+ 516us) server_negotiation.cc:299] Negotiation successful
0620 06:06:01.642987 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":135,"threads_started":1}
I20260620 06:06:01.643962 21714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60bee7dfe01f43ad94bdc847ee03ba45. Latest consensus state: current_term: 1 leader_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45007 } } }
I20260620 06:06:01.644101 21714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:01.644410 21715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60bee7dfe01f43ad94bdc847ee03ba45" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 45007 } } }
I20260620 06:06:01.644493 21722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:01.644516 21715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:01.648396 21722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:01.654656 21722 catalog_manager.cc:1347] Generated new cluster ID: d5333f512be0444cb9013659c136d908
I20260620 06:06:01.654776 21722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:01.671535 21722 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:01.672637 21722 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:01.684151 21722 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45: Generated new TSK 0
I20260620 06:06:01.685146 21722 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:06:01.714308 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45007
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:01.830751 21735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:01.831049 21735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:01.831100 21735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:01.835738 21735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:01.835811 21735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:01.835911 21735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:06:01.841670 21735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:01.842866 21735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:01.844089 21735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:01.854578 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:01.854578 21743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:01.854724 21735 server_base.cc:1034] running on GCE node
W20260620 06:06:01.854900 21740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:01.855336 21735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:01.856060 21735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:01.857239 21735 hybrid_clock.cc:648] HybridClock initialized: now 1781935561857204 us; error 52 us; skew 500 ppm
Jun 20 06:06:01 dist-test-slave-4h14 krb5kdc[21628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935561, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:01.865309 21735 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:06:01.866324 21735 webserver.cc:460] Webserver started at http://127.3.115.1:35199/ using document root <none> and password file <none>
I20260620 06:06:01.866940 21735 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:01.867031 21735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:01.867274 21735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:01.869529 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "0600a52a02ff478a913a371a1e343ca6"
format_stamp: "Formatted at 2026-06-20 06:06:01 on dist-test-slave-4h14"
server_key: "f133ec1ca3d909d186eaf1159eae331c"
server_key_iv: "8d3c409144b1ec97af9cc46a5b1f18bf"
server_key_version: "encryptionkey@0"
I20260620 06:06:01.870117 21735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "0600a52a02ff478a913a371a1e343ca6"
format_stamp: "Formatted at 2026-06-20 06:06:01 on dist-test-slave-4h14"
server_key: "f133ec1ca3d909d186eaf1159eae331c"
server_key_iv: "8d3c409144b1ec97af9cc46a5b1f18bf"
server_key_version: "encryptionkey@0"
I20260620 06:06:01.874012 21735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:01.876624 21750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:01.878094 21735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:06:01.878258 21735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "0600a52a02ff478a913a371a1e343ca6"
format_stamp: "Formatted at 2026-06-20 06:06:01 on dist-test-slave-4h14"
server_key: "f133ec1ca3d909d186eaf1159eae331c"
server_key_iv: "8d3c409144b1ec97af9cc46a5b1f18bf"
server_key_version: "encryptionkey@0"
I20260620 06:06:01.878393 21735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:01.901074 21735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:01.904265 21735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:01.904479 21735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:01.905141 21735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:01.906008 21735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:01.906054 21735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:01.906122 21735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:01.906152 21735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:01.915962 21735 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35605
I20260620 06:06:01.916006 21863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35605 every 8 connection(s)
I20260620 06:06:01.917022 21735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:01.921756 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21735
I20260620 06:06:01.921900 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:01.922176 3532 external_mini_cluster.cc:1428] Setting key db19c63689f323fbacc0db3fb4841936
Jun 20 06:06:01 dist-test-slave-4h14 krb5kdc[21628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935561, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:06:01.933389 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:01.919010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:33805 (local address 127.3.115.62:45007)
0620 06:06:01.919155 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:01.919159 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:01.920352 (+ 1193us) server_negotiation.cc:407] Connection header received
0620 06:06:01.921414 (+ 1062us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:01.921417 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:01.921481 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:01.921589 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:01.923189 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:01.923936 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:01.925350 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:01.925643 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:01.925736 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:01.929363 (+ 3627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:01.929366 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:01.929368 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:01.929395 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:01.931080 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:01.931701 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:01.931705 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:01.931706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:01.931791 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:01.932147 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:01.932155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:01.932157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:01.932356 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:01.932462 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:01.933085 (+ 623us) server_negotiation.cc:299] Negotiation successful
0620 06:06:01.933221 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:06:01.934463 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:01.919450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45007 (local address 127.3.115.1:33805)
0620 06:06:01.920183 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:01.920229 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:01.921161 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:01.921780 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:01.921788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:01.922293 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:01.923004 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:01.923025 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:01.924087 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:01.924090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:01.925133 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:01.925163 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:01.925831 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:01.925838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:01.926092 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:01.926803 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:01.926840 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:01.929131 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:01.931240 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:01.931247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:01.931273 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:01.931584 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:01.931893 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:01.931896 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:01.931898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:01.932050 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:01.932460 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:01.932468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:01.932838 (+ 370us) client_negotiation.cc:769] Sending connection context
0620 06:06:01.933031 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 06:06:01.933403 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":116,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:06:01.936276 21864 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45007
I20260620 06:06:01.936583 21864 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:01.937150 21864 heartbeater.cc:511] Master 127.3.115.62:45007 requested a full tablet report, sending...
I20260620 06:06:01.938876 21673 ts_manager.cc:194] Registered new tserver with Master: 0600a52a02ff478a913a371a1e343ca6 (127.3.115.1:35605)
I20260620 06:06:01.940573 21673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:33805
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:06:01.952309 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45007
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:02.070108 21869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:02.070449 21869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:02.070531 21869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:02.074823 21869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:02.074935 21869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:02.075042 21869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:06:02.080858 21869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:02.082094 21869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:02.083283 21869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:02.093207 21874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:02.093211 21877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:02.093200 21875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:02.093305 21869 server_base.cc:1034] running on GCE node
I20260620 06:06:02.093820 21869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:02.094529 21869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:02.095718 21869 hybrid_clock.cc:648] HybridClock initialized: now 1781935562095691 us; error 41 us; skew 500 ppm
Jun 20 06:06:02 dist-test-slave-4h14 krb5kdc[21628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935562, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:02.098533 21869 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:06:02.099493 21869 webserver.cc:460] Webserver started at http://127.3.115.2:42133/ using document root <none> and password file <none>
I20260620 06:06:02.100080 21869 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:02.100168 21869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:02.100404 21869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:02.102607 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "96998a40395c41158c85e5c64bd3c006"
format_stamp: "Formatted at 2026-06-20 06:06:02 on dist-test-slave-4h14"
server_key: "895463396aa25ab7e429aafdd87690f7"
server_key_iv: "642e16c57b4b61f0df1d894f90e5f3c1"
server_key_version: "encryptionkey@0"
I20260620 06:06:02.103205 21869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "96998a40395c41158c85e5c64bd3c006"
format_stamp: "Formatted at 2026-06-20 06:06:02 on dist-test-slave-4h14"
server_key: "895463396aa25ab7e429aafdd87690f7"
server_key_iv: "642e16c57b4b61f0df1d894f90e5f3c1"
server_key_version: "encryptionkey@0"
I20260620 06:06:02.107162 21869 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:02.109781 21884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:02.111094 21869 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:02.111240 21869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "96998a40395c41158c85e5c64bd3c006"
format_stamp: "Formatted at 2026-06-20 06:06:02 on dist-test-slave-4h14"
server_key: "895463396aa25ab7e429aafdd87690f7"
server_key_iv: "642e16c57b4b61f0df1d894f90e5f3c1"
server_key_version: "encryptionkey@0"
I20260620 06:06:02.111366 21869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:02.124044 21869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:02.127414 21869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:02.127640 21869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:02.128278 21869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:02.129211 21869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:02.129283 21869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:02.129359 21869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:02.129398 21869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:02.138967 21869 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:45843
I20260620 06:06:02.139020 21997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:45843 every 8 connection(s)
I20260620 06:06:02.139971 21869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:02.149577 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 21869
I20260620 06:06:02.149732 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:02.150118 3532 external_mini_cluster.cc:1428] Setting key a37e49134088709dce0380d7f25cbadd
Jun 20 06:06:02 dist-test-slave-4h14 krb5kdc[21628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935562, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:02.155827 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.141988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:50517 (local address 127.3.115.62:45007)
0620 06:06:02.142123 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.142127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.143104 (+ 977us) server_negotiation.cc:407] Connection header received
0620 06:06:02.144115 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.144119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.144182 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.144386 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:02.145872 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.146445 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.147782 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.148008 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.148108 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.151612 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:02.151624 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:02.151627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:02.151656 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:02.153465 (+ 1809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:02.154020 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:02.154024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:02.154026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:02.154116 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:02.154479 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:02.154484 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:02.154486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:02.154695 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:02.154797 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.155499 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.155657 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:06:02.157001 22000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.142393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45007 (local address 127.3.115.2:50517)
0620 06:06:02.142919 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.142966 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.143877 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.144591 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.144603 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.145078 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:02.145705 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.145722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.146601 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.146604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.147641 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.147651 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.148210 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.148213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.148384 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.149181 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:02.149203 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:02.151411 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:02.153596 (+ 2185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:02.153603 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:02.153621 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:02.153910 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:02.154214 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:02.154217 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:02.154219 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:02.154382 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:02.154813 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:02.154822 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:02.155109 (+ 287us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.155445 (+ 336us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.156013 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":121,"threads_started":1}
I20260620 06:06:02.158339 21998 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45007
I20260620 06:06:02.158604 21998 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:02.159200 21998 heartbeater.cc:511] Master 127.3.115.62:45007 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
I20260620 06:06:02.160346 21673 ts_manager.cc:194] Registered new tserver with Master: 96998a40395c41158c85e5c64bd3c006 (127.3.115.2:45843)
I20260620 06:06:02.161132 21673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:50517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:06:02.179760 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45007
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:02.294399 22003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:02.294698 22003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:02.294744 22003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:02.298902 22003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:02.298982 22003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:02.299045 22003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:06:02.304649 22003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:45007
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:02.305802 22003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:02.306823 22003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:02.316825 22009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:02.316825 22011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:02.316826 22008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:02.317188 22003 server_base.cc:1034] running on GCE node
I20260620 06:06:02.317619 22003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:02.318256 22003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:02.319451 22003 hybrid_clock.cc:648] HybridClock initialized: now 1781935562319431 us; error 33 us; skew 500 ppm
Jun 20 06:06:02 dist-test-slave-4h14 krb5kdc[21628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935562, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:02.322414 22003 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:06:02.323385 22003 webserver.cc:460] Webserver started at http://127.3.115.3:33561/ using document root <none> and password file <none>
I20260620 06:06:02.323992 22003 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:02.324056 22003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:02.324272 22003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:02.326525 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "641eb9a13e054a7c8f34c1ba6e6b953c"
format_stamp: "Formatted at 2026-06-20 06:06:02 on dist-test-slave-4h14"
server_key: "2aaad697e1c6dbc95db16fb52a4d9c3b"
server_key_iv: "6c7e285243dce784739894d656ddc5df"
server_key_version: "encryptionkey@0"
I20260620 06:06:02.327138 22003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "641eb9a13e054a7c8f34c1ba6e6b953c"
format_stamp: "Formatted at 2026-06-20 06:06:02 on dist-test-slave-4h14"
server_key: "2aaad697e1c6dbc95db16fb52a4d9c3b"
server_key_iv: "6c7e285243dce784739894d656ddc5df"
server_key_version: "encryptionkey@0"
I20260620 06:06:02.331408 22003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:06:02.334199 22018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:02.335629 22003 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:02.335776 22003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "641eb9a13e054a7c8f34c1ba6e6b953c"
format_stamp: "Formatted at 2026-06-20 06:06:02 on dist-test-slave-4h14"
server_key: "2aaad697e1c6dbc95db16fb52a4d9c3b"
server_key_iv: "6c7e285243dce784739894d656ddc5df"
server_key_version: "encryptionkey@0"
I20260620 06:06:02.335902 22003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:02.355991 22003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:02.359290 22003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:02.359512 22003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:02.360147 22003 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:02.361140 22003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:02.361189 22003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:02.361231 22003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:02.361248 22003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:02.370775 22003 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:40185
I20260620 06:06:02.370793 22131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:40185 every 8 connection(s)
I20260620 06:06:02.371791 22003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:02.377074 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22003
I20260620 06:06:02.377189 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:02.377496 3532 external_mini_cluster.cc:1428] Setting key 0080fcbdcbecf1e3779b459f0067b611
Jun 20 06:06:02 dist-test-slave-4h14 krb5kdc[21628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935562, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:02.386024 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.373789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:57613 (local address 127.3.115.62:45007)
0620 06:06:02.373930 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.373934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.374835 (+ 901us) server_negotiation.cc:407] Connection header received
0620 06:06:02.375810 (+ 975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.375813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.375878 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.376010 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:02.377402 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.378066 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.379482 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.379794 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.379910 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.382504 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:02.382508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:02.382511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:02.382541 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:02.383914 (+ 1373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:02.384439 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:02.384443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:02.384445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:02.384527 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:02.384895 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:02.384900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:02.384902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:02.385091 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:02.385206 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.385719 (+ 513us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.385855 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:02.386855 22134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.374165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:45007 (local address 127.3.115.3:57613)
0620 06:06:02.374663 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.374711 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.375580 (+ 869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.376187 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.376194 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.376590 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:02.377162 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.377183 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.378193 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.378196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.379362 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.379372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.379954 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.379957 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.380136 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.380752 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:02.380769 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:02.382333 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:02.384039 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:02.384046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:02.384063 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:02.384336 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:02.384622 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:02.384625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:02.384627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:02.384802 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:02.385203 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:02.385209 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:02.385480 (+ 271us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.385665 (+ 185us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.386005 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":122,"threads_started":1}
I20260620 06:06:02.387979 22132 heartbeater.cc:347] Connected to a master server at 127.3.115.62:45007
I20260620 06:06:02.388204 22132 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:02.388787 22132 heartbeater.cc:511] Master 127.3.115.62:45007 requested a full tablet report, sending...
I20260620 06:06:02.389943 21673 ts_manager.cc:194] Registered new tserver with Master: 641eb9a13e054a7c8f34c1ba6e6b953c (127.3.115.3:40185)
I20260620 06:06:02.390727 21673 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:57613
I20260620 06:06:02.391505 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:06:02.401552 21712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.393585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56086 (local address 127.3.115.62:45007)
0620 06:06:02.393750 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.393754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.393982 (+ 228us) server_negotiation.cc:407] Connection header received
0620 06:06:02.394056 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.394059 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.394100 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.394198 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:02.394864 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.395385 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.396209 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.396420 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.396502 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.398183 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:02.398186 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:02.398189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:02.398212 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:02.399715 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:02.400236 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:02.400238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:02.400239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:02.400323 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:02.400602 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:02.400604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:02.400605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:02.400890 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:02.401015 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.401262 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.401382 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260620 06:06:02.404932 21673 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56086:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:02.408116 21673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:02.424136 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.418545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52426 (local address 127.3.115.3:40185)
0620 06:06:02.419362 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.419372 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.419389 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:06:02.419449 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.419453 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.419598 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.420059 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:02.420215 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.421008 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.422505 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.423347 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.423669 (+ 322us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.423789 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.423857 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.423963 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":119,"threads_started":1}
I20260620 06:06:02.424127 22143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.417734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40185 (local address 127.0.0.1:52426)
0620 06:06:02.418367 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.418380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.418455 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.419775 (+ 1320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.419778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.419798 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:02.419904 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.419912 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.421143 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.421147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.422365 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.422375 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.423494 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.423498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.423687 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.423705 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.423783 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.423963 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":133,"thread_start_us":94,"threads_started":1}
I20260620 06:06:02.424216 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.417851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.3.115.1:35605)
0620 06:06:02.418680 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.418696 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.418722 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:06:02.418808 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.418813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.419059 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.419584 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:02.419763 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.420892 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.422645 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.423289 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.423413 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.423570 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.423965 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.424062 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":144,"threads_started":1}
I20260620 06:06:02.424479 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.418484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:45843 (local address 127.0.0.1:60672)
0620 06:06:02.418723 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.418737 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.418819 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.419921 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.419924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.419941 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:02.420024 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.420030 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.421437 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.421441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.422633 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.422640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.423834 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.423837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.423925 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.423948 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.424019 (+ 71us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.424090 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":42,"thread_start_us":97,"threads_started":1}
I20260620 06:06:02.424642 22144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.417709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35605 (local address 127.0.0.1:46614)
0620 06:06:02.418168 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.418227 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.418366 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.419240 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.419247 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.419272 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:02.419414 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.419422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.421046 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.421050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.422500 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.422511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.423411 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.423414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.423577 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.423675 (+ 98us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.423792 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.424502 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":675,"thread_start_us":84,"threads_started":1}
I20260620 06:06:02.424857 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.418902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60672 (local address 127.3.115.2:45843)
0620 06:06:02.419289 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.419303 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.419329 (+ 26us) server_negotiation.cc:407] Connection header received
0620 06:06:02.419409 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.419416 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.419599 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.419759 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:02.420155 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.421299 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.422747 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.423527 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.423625 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.423771 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.424200 (+ 429us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.424705 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":153,"threads_started":1}
I20260620 06:06:02.427361 21798 tablet_service.cc:1467] Processing CreateTablet for tablet 59be06af7d7647c8b6acedcb37c2fb92 (DEFAULT_TABLE table=test-table [id=1d08360f1079475a98fc0bf925c874f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:02.427490 21932 tablet_service.cc:1467] Processing CreateTablet for tablet 59be06af7d7647c8b6acedcb37c2fb92 (DEFAULT_TABLE table=test-table [id=1d08360f1079475a98fc0bf925c874f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:02.427490 22066 tablet_service.cc:1467] Processing CreateTablet for tablet 59be06af7d7647c8b6acedcb37c2fb92 (DEFAULT_TABLE table=test-table [id=1d08360f1079475a98fc0bf925c874f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:02.428426 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59be06af7d7647c8b6acedcb37c2fb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:02.428525 21932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59be06af7d7647c8b6acedcb37c2fb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:02.428525 22066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59be06af7d7647c8b6acedcb37c2fb92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:02.435223 22149 tablet_bootstrap.cc:492] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: Bootstrap starting.
I20260620 06:06:02.436486 22150 tablet_bootstrap.cc:492] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Bootstrap starting.
I20260620 06:06:02.436597 22151 tablet_bootstrap.cc:492] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Bootstrap starting.
I20260620 06:06:02.437929 22149 tablet_bootstrap.cc:654] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:02.438304 22150 tablet_bootstrap.cc:654] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:02.438704 22151 tablet_bootstrap.cc:654] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:02.438853 22149 log.cc:824] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:02.439486 22150 log.cc:824] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:02.439868 22151 log.cc:824] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:02.441354 22149 tablet_bootstrap.cc:492] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: No bootstrap required, opened a new log
I20260620 06:06:02.441562 22149 ts_tablet_manager.cc:1397] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:06:02.441684 22150 tablet_bootstrap.cc:492] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: No bootstrap required, opened a new log
I20260620 06:06:02.441761 22151 tablet_bootstrap.cc:492] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: No bootstrap required, opened a new log
I20260620 06:06:02.441877 22150 ts_tablet_manager.cc:1397] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:02.441932 22151 ts_tablet_manager.cc:1397] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:06:02.445149 22150 raft_consensus.cc:348] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.445420 22150 raft_consensus.cc:374] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:02.445482 22150 raft_consensus.cc:729] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96998a40395c41158c85e5c64bd3c006, State: Initialized, Role: FOLLOWER
I20260620 06:06:02.445921 22150 consensus_queue.cc:260] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.445873 22151 raft_consensus.cc:348] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.446143 22151 raft_consensus.cc:374] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:02.446209 22151 raft_consensus.cc:729] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641eb9a13e054a7c8f34c1ba6e6b953c, State: Initialized, Role: FOLLOWER
I20260620 06:06:02.446131 22149 raft_consensus.cc:348] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.446379 22149 raft_consensus.cc:374] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:02.446437 22149 raft_consensus.cc:729] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0600a52a02ff478a913a371a1e343ca6, State: Initialized, Role: FOLLOWER
I20260620 06:06:02.446684 22151 consensus_queue.cc:260] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.446789 22149 consensus_queue.cc:260] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.447011 22150 ts_tablet_manager.cc:1428] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Time spent starting tablet: real 0.005s user 0.003s sys 0.003s
I20260620 06:06:02.447379 21998 heartbeater.cc:503] Master 127.3.115.62:45007 was elected leader, sending a full tablet report...
I20260620 06:06:02.447679 22149 ts_tablet_manager.cc:1428] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:06:02.447679 22151 ts_tablet_manager.cc:1428] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260620 06:06:02.447996 21864 heartbeater.cc:503] Master 127.3.115.62:45007 was elected leader, sending a full tablet report...
I20260620 06:06:02.448959 22132 heartbeater.cc:503] Master 127.3.115.62:45007 was elected leader, sending a full tablet report...
W20260620 06:06:02.622833 22133 tablet.cc:2369] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:06:02.641171 21999 tablet.cc:2369] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 06:06:02.668581 21865 tablet.cc:2369] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:02.758632 22157 raft_consensus.cc:482] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:02.758852 22157 raft_consensus.cc:504] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.760198 22157 leader_election.cc:288] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 641eb9a13e054a7c8f34c1ba6e6b953c (127.3.115.3:40185), 96998a40395c41158c85e5c64bd3c006 (127.3.115.2:45843)
I20260620 06:06:02.764348 22147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.760560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:42259 (local address 127.3.115.3:40185)
0620 06:06:02.760741 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.760746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.760997 (+ 251us) server_negotiation.cc:407] Connection header received
0620 06:06:02.761187 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.761189 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.761234 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.761323 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:02.761621 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.762129 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.763289 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.763760 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.763879 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.763962 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.764134 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.764218 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:02.764348 22148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.760723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:34669 (local address 127.3.115.2:45843)
0620 06:06:02.760869 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.760874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.761037 (+ 163us) server_negotiation.cc:407] Connection header received
0620 06:06:02.761197 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.761201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.761243 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.761337 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:02.761632 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.762208 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.763309 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.763712 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.763843 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.763912 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.764112 (+ 200us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.764190 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260620 06:06:02.764357 22159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.760559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:45843 (local address 127.3.115.1:34669)
0620 06:06:02.760928 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.760940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.761075 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.761347 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.761349 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.761365 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:02.761516 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.761522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.762346 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.762349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.763172 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.763182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.763869 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.763872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.763982 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.764002 (+ 20us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.764093 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.764189 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":170,"threads_started":1}
I20260620 06:06:02.764361 22158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.760414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:40185 (local address 127.3.115.1:42259)
0620 06:06:02.760881 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.760898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:02.761052 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:02.761338 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:02.761341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:02.761357 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:02.761500 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.761505 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.762260 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.762264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.763157 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:02.763167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.763885 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.763888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:02.764005 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.764027 (+ 22us) client_negotiation.cc:769] Sending connection context
0620 06:06:02.764117 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:06:02.764195 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":138,"threads_started":1}
I20260620 06:06:02.765148 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be06af7d7647c8b6acedcb37c2fb92" candidate_uuid: "0600a52a02ff478a913a371a1e343ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" is_pre_election: true
I20260620 06:06:02.765148 21952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be06af7d7647c8b6acedcb37c2fb92" candidate_uuid: "0600a52a02ff478a913a371a1e343ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96998a40395c41158c85e5c64bd3c006" is_pre_election: true
I20260620 06:06:02.765529 21952 raft_consensus.cc:2454] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0600a52a02ff478a913a371a1e343ca6 in term 0.
I20260620 06:06:02.765529 22086 raft_consensus.cc:2454] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0600a52a02ff478a913a371a1e343ca6 in term 0.
I20260620 06:06:02.766232 21751 leader_election.cc:302] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0600a52a02ff478a913a371a1e343ca6, 641eb9a13e054a7c8f34c1ba6e6b953c, 96998a40395c41158c85e5c64bd3c006; no voters:
I20260620 06:06:02.766582 22157 raft_consensus.cc:2781] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:02.766695 22157 raft_consensus.cc:482] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:02.766741 22157 raft_consensus.cc:3037] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:02.767845 22157 raft_consensus.cc:504] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.768241 22157 leader_election.cc:288] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [CANDIDATE]: Term 1 election: Requested vote from peers 641eb9a13e054a7c8f34c1ba6e6b953c (127.3.115.3:40185), 96998a40395c41158c85e5c64bd3c006 (127.3.115.2:45843)
I20260620 06:06:02.768746 22086 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be06af7d7647c8b6acedcb37c2fb92" candidate_uuid: "0600a52a02ff478a913a371a1e343ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c"
I20260620 06:06:02.768862 21952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59be06af7d7647c8b6acedcb37c2fb92" candidate_uuid: "0600a52a02ff478a913a371a1e343ca6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96998a40395c41158c85e5c64bd3c006"
I20260620 06:06:02.768905 22086 raft_consensus.cc:3037] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:02.769032 21952 raft_consensus.cc:3037] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:02.770125 21952 raft_consensus.cc:2454] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0600a52a02ff478a913a371a1e343ca6 in term 1.
I20260620 06:06:02.770150 22086 raft_consensus.cc:2454] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0600a52a02ff478a913a371a1e343ca6 in term 1.
I20260620 06:06:02.770550 21752 leader_election.cc:302] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0600a52a02ff478a913a371a1e343ca6, 641eb9a13e054a7c8f34c1ba6e6b953c; no voters:
I20260620 06:06:02.770776 22157 raft_consensus.cc:2781] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:02.771042 22157 raft_consensus.cc:686] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 1 LEADER]: Becoming Leader. State: Replica: 0600a52a02ff478a913a371a1e343ca6, State: Running, Role: LEADER
I20260620 06:06:02.771613 22157 consensus_queue.cc:237] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } }
I20260620 06:06:02.775692 21673 catalog_manager.cc:5495] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0600a52a02ff478a913a371a1e343ca6 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "0600a52a02ff478a913a371a1e343ca6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0600a52a02ff478a913a371a1e343ca6" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 } health_report { overall_health: UNKNOWN } } }
I20260620 06:06:02.791014 22145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.786916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46622 (local address 127.3.115.1:35605)
0620 06:06:02.787079 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:02.787083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:02.787125 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:06:02.787201 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:02.787203 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:02.787262 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:02.787370 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:02.787707 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.788230 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.789337 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:02.789577 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:02.789686 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:02.790039 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:02.790479 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:02.790604 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:02.790844 (+ 240us) server_negotiation.cc:299] Negotiation successful
0620 06:06:02.790889 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 06:06:02.798113 22086 raft_consensus.cc:1261] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 1 FOLLOWER]: Refusing update from remote peer 0600a52a02ff478a913a371a1e343ca6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:02.798103 21952 raft_consensus.cc:1261] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 1 FOLLOWER]: Refusing update from remote peer 0600a52a02ff478a913a371a1e343ca6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:02.798936 22160 consensus_queue.cc:1035] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "641eb9a13e054a7c8f34c1ba6e6b953c" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 40185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:02.799175 22157 consensus_queue.cc:1035] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96998a40395c41158c85e5c64bd3c006" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 45843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:02.809376 22164 mvcc.cc:205] Tried to move back new op lower bound from 7298808065212768256 to 7298808065117388800. Current Snapshot: MvccSnapshot[applied={T|T < 7298808065212768256}]
I20260620 06:06:02.817912 21671 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:02.818086 21671 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56086:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:02.820916 21671 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60bee7dfe01f43ad94bdc847ee03ba45: Sending DeleteTablet for 3 replicas of tablet 59be06af7d7647c8b6acedcb37c2fb92
I20260620 06:06:02.822116 21798 tablet_service.cc:1514] Processing DeleteTablet for tablet 59be06af7d7647c8b6acedcb37c2fb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:02 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:46614
I20260620 06:06:02.822119 22066 tablet_service.cc:1514] Processing DeleteTablet for tablet 59be06af7d7647c8b6acedcb37c2fb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:02 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:52426
I20260620 06:06:02.822734 22173 tablet_replica.cc:330] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: stopping tablet replica
I20260620 06:06:02.822885 21932 tablet_service.cc:1514] Processing DeleteTablet for tablet 59be06af7d7647c8b6acedcb37c2fb92 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:02 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:60672
I20260620 06:06:02.823243 22173 raft_consensus.cc:2229] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:02.823493 22174 tablet_replica.cc:330] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6: stopping tablet replica
I20260620 06:06:02.823588 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21735
I20260620 06:06:02.823850 22174 raft_consensus.cc:2229] T 59be06af7d7647c8b6acedcb37c2fb92 P 0600a52a02ff478a913a371a1e343ca6 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:02.825317 22175 tablet_replica.cc:330] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: stopping tablet replica
I20260620 06:06:02.825477 22173 raft_consensus.cc:2258] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:02.825695 22175 raft_consensus.cc:2229] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:02.826509 22175 raft_consensus.cc:2258] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:02.827330 22173 ts_tablet_manager.cc:1905] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:02.828022 22175 ts_tablet_manager.cc:1905] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:02.830392 22175 ts_tablet_manager.cc:1918] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:02.830528 22175 log.cc:1196] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/59be06af7d7647c8b6acedcb37c2fb92
I20260620 06:06:02.830812 22175 ts_tablet_manager.cc:1939] T 59be06af7d7647c8b6acedcb37c2fb92 P 96998a40395c41158c85e5c64bd3c006: Deleting consensus metadata
I20260620 06:06:02.831061 22173 ts_tablet_manager.cc:1918] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:02.831187 22173 log.cc:1196] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/59be06af7d7647c8b6acedcb37c2fb92
I20260620 06:06:02.831548 22173 ts_tablet_manager.cc:1939] T 59be06af7d7647c8b6acedcb37c2fb92 P 641eb9a13e054a7c8f34c1ba6e6b953c: Deleting consensus metadata
W20260620 06:06:02.831532 21657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:35605 (error 108)
I20260620 06:06:02.832261 21657 catalog_manager.cc:4841] TS 96998a40395c41158c85e5c64bd3c006 (127.3.115.2:45843): tablet 59be06af7d7647c8b6acedcb37c2fb92 (table test-table [id=1d08360f1079475a98fc0bf925c874f1]) successfully deleted
I20260620 06:06:02.832435 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21869
I20260620 06:06:02.834156 21658 catalog_manager.cc:4841] TS 641eb9a13e054a7c8f34c1ba6e6b953c (127.3.115.3:40185): tablet 59be06af7d7647c8b6acedcb37c2fb92 (table test-table [id=1d08360f1079475a98fc0bf925c874f1]) successfully deleted
I20260620 06:06:02.834754 22146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:02.834474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35605 (local address 127.0.0.1:46624)
0620 06:06:02.834605 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:02.834683 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260620 06:06:02.834937 21657 catalog_manager.cc:4584] TS 0600a52a02ff478a913a371a1e343ca6 (127.3.115.1:35605): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59be06af7d7647c8b6acedcb37c2fb92: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35605: connect: Connection refused (error 111)
I20260620 06:06:02.838462 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22003
I20260620 06:06:02.844442 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 21641
2026-06-20T06:06:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2590 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T06:06:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:02Z Disabled control of system clock
I20260620 06:06:02.875104 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:41303
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:41303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:06:02.989670 22180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:02.990031 22180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:02.990072 22180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:02.994153 22180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:02.994225 22180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:02.994240 22180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:02.994257 22180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:02.994274 22180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:02.999191 22180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:41303
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:41303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:03.000458 22180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:03.002039 22180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:03.011116 22188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.011111 22185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.011114 22186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:03.011528 22180 server_base.cc:1034] running on GCE node
I20260620 06:06:03.012138 22180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:03.013351 22180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:03.014515 22180 hybrid_clock.cc:648] HybridClock initialized: now 1781935563014500 us; error 30 us; skew 500 ppm
I20260620 06:06:03.016539 22180 webserver.cc:460] Webserver started at http://127.3.115.62:40757/ using document root <none> and password file <none>
I20260620 06:06:03.017174 22180 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:03.017239 22180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:03.017467 22180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:03.019665 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "5d69f3578cfc42e197f61f324b1148b8"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "17a00b24a05e1522255b99d954c50cdf"
server_key_iv: "43c0f71231cdbfe7224016046a510d7c"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.020236 22180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "5d69f3578cfc42e197f61f324b1148b8"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "17a00b24a05e1522255b99d954c50cdf"
server_key_iv: "43c0f71231cdbfe7224016046a510d7c"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.024000 22180 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260620 06:06:03.026683 22194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.028074 22180 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260620 06:06:03.028213 22180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "5d69f3578cfc42e197f61f324b1148b8"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "17a00b24a05e1522255b99d954c50cdf"
server_key_iv: "43c0f71231cdbfe7224016046a510d7c"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.028342 22180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:03.039028 22180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:03.039825 22180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:03.040045 22180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:03.047720 22180 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:41303
I20260620 06:06:03.047753 22246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:41303 every 8 connection(s)
I20260620 06:06:03.048768 22180 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:03.051391 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22180
I20260620 06:06:03.051555 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:03.051888 3532 external_mini_cluster.cc:1428] Setting key 3d8a210e8a743f080f71b3f37eef26f5
I20260620 06:06:03.052117 22247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:03.059986 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8: Bootstrap starting.
I20260620 06:06:03.063216 22247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:03.063252 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.053487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33008 (local address 127.3.115.62:41303)
0620 06:06:03.054251 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.054271 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.054317 (+ 46us) server_negotiation.cc:407] Connection header received
0620 06:06:03.055263 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.055289 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.055836 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.056238 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:03.056743 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.058183 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.059076 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.059763 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.059928 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.060938 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:03.060945 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:03.060960 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:03.061501 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:03.061576 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:03.061723 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:03.061840 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.061986 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.062270 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":55040,"thread_start_us":120,"threads_started":1}
I20260620 06:06:03.064162 22247 log.cc:824] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:03.067142 22247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8: No bootstrap required, opened a new log
I20260620 06:06:03.071198 22247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69f3578cfc42e197f61f324b1148b8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41303 } }
I20260620 06:06:03.071467 22247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:03.071539 22247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d69f3578cfc42e197f61f324b1148b8, State: Initialized, Role: FOLLOWER
I20260620 06:06:03.071997 22247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69f3578cfc42e197f61f324b1148b8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41303 } }
I20260620 06:06:03.072132 22247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:03.072182 22247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:03.072254 22247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:03.073231 22247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69f3578cfc42e197f61f324b1148b8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41303 } }
I20260620 06:06:03.073531 22247 leader_election.cc:302] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d69f3578cfc42e197f61f324b1148b8; no voters:
I20260620 06:06:03.073817 22247 leader_election.cc:288] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:03.073932 22252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:03.074186 22252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [term 1 LEADER]: Becoming Leader. State: Replica: 5d69f3578cfc42e197f61f324b1148b8, State: Running, Role: LEADER
I20260620 06:06:03.074548 22252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69f3578cfc42e197f61f324b1148b8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41303 } }
I20260620 06:06:03.075104 22247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:03.076287 22253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d69f3578cfc42e197f61f324b1148b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69f3578cfc42e197f61f324b1148b8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41303 } } }
I20260620 06:06:03.076272 22254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d69f3578cfc42e197f61f324b1148b8. Latest consensus state: current_term: 1 leader_uuid: "5d69f3578cfc42e197f61f324b1148b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d69f3578cfc42e197f61f324b1148b8" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41303 } } }
I20260620 06:06:03.076439 22253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:03.076440 22254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:03.076848 22261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:03.079810 22261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:03.085722 22261 catalog_manager.cc:1347] Generated new cluster ID: dad1aa5f600d4aad9f9bea7ad11a17bf
I20260620 06:06:03.085794 22261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:03.097961 22261 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:03.098955 22261 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:03.106590 22261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8: Generated new TSK 0
I20260620 06:06:03.107374 22261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:03.118814 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41303
--builtin_ntp_servers=127.3.115.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:06:03.236259 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:03.236552 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:03.236590 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:03.240940 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:03.241010 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:03.241070 22271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:06:03.246902 22271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:03.248580 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:03.250415 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:03.261183 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.261245 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.261183 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:03.261420 22271 server_base.cc:1034] running on GCE node
I20260620 06:06:03.261909 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:03.262638 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:03.263839 22271 hybrid_clock.cc:648] HybridClock initialized: now 1781935563263815 us; error 38 us; skew 500 ppm
I20260620 06:06:03.266010 22271 webserver.cc:460] Webserver started at http://127.3.115.1:36361/ using document root <none> and password file <none>
I20260620 06:06:03.266645 22271 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:03.266712 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:03.266939 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:03.269207 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "8516eab892534d6280681fe31621d20a"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "8278103969af46d47ff495140b5a0f99"
server_key_iv: "a43dd3c6fdf6b7f675d76f29fe102ac6"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.269762 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "8516eab892534d6280681fe31621d20a"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "8278103969af46d47ff495140b5a0f99"
server_key_iv: "a43dd3c6fdf6b7f675d76f29fe102ac6"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.273907 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:03.276999 22285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.278463 22271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:06:03.278577 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "8516eab892534d6280681fe31621d20a"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "8278103969af46d47ff495140b5a0f99"
server_key_iv: "a43dd3c6fdf6b7f675d76f29fe102ac6"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.278728 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:03.301463 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:03.302214 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:03.302426 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:03.303073 22271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:03.304055 22271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:03.304102 22271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.304168 22271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:03.304198 22271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.313261 22271 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:44915
I20260620 06:06:03.313282 22398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:44915 every 8 connection(s)
I20260620 06:06:03.314189 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:03.315806 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22271
I20260620 06:06:03.315932 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:03.316288 3532 external_mini_cluster.cc:1428] Setting key a8523a1343856cfe55debf3e217025b3
I20260620 06:06:03.318552 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41303
--builtin_ntp_servers=127.3.115.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:06:03.325991 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.316235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:53973 (local address 127.3.115.62:41303)
0620 06:06:03.316392 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.316397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.317640 (+ 1243us) server_negotiation.cc:407] Connection header received
0620 06:06:03.319718 (+ 2078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.319722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.319778 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.319861 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:03.320972 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.321607 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.323536 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.323759 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.323847 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.325406 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:03.325410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:03.325412 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:03.325426 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:03.325448 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:03.325457 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:03.325531 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.325772 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.325853 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 06:06:03.327390 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.316728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:41303 (local address 127.3.115.1:53973)
0620 06:06:03.317487 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.317528 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.319446 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.320100 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.320113 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.320234 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:03.320826 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.320841 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.321770 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.321776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.323401 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.323416 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.323932 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.323938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.324194 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.325002 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:06:03.325005 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:03.325088 (+ 83us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:06:03.325198 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:03.325555 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:03.325568 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:03.325576 (+ 8us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.325734 (+ 158us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.326129 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
I20260620 06:06:03.328814 22399 heartbeater.cc:347] Connected to a master server at 127.3.115.62:41303
I20260620 06:06:03.329071 22399 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:03.329667 22399 heartbeater.cc:511] Master 127.3.115.62:41303 requested a full tablet report, sending...
I20260620 06:06:03.331238 22211 ts_manager.cc:194] Registered new tserver with Master: 8516eab892534d6280681fe31621d20a (127.3.115.1:44915)
I20260620 06:06:03.332527 22211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.1:53973
W20260620 06:06:03.433917 22402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:03.434263 22402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:03.434347 22402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:03.438660 22402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:03.438721 22402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:03.438781 22402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:06:03.443672 22402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:03.444881 22402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:03.446270 22402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:03.456346 22407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.456346 22408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:03.456440 22402 server_base.cc:1034] running on GCE node
W20260620 06:06:03.456808 22410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:03.457309 22402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:03.457999 22402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:03.459184 22402 hybrid_clock.cc:648] HybridClock initialized: now 1781935563459130 us; error 67 us; skew 500 ppm
I20260620 06:06:03.461123 22402 webserver.cc:460] Webserver started at http://127.3.115.2:40505/ using document root <none> and password file <none>
I20260620 06:06:03.461716 22402 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:03.461781 22402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:03.462031 22402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:03.464244 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "0bd1457342fd479eb457c55e8a533ef8"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "5b406c0d086bd109a506b3f3a6b53f6f"
server_key_iv: "b5dd39951f7388e4ba9d53185e50848f"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.464923 22402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "0bd1457342fd479eb457c55e8a533ef8"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "5b406c0d086bd109a506b3f3a6b53f6f"
server_key_iv: "b5dd39951f7388e4ba9d53185e50848f"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.469007 22402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:06:03.471621 22416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.473012 22402 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:03.473141 22402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "0bd1457342fd479eb457c55e8a533ef8"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "5b406c0d086bd109a506b3f3a6b53f6f"
server_key_iv: "b5dd39951f7388e4ba9d53185e50848f"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.473259 22402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:03.488842 22402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:03.489605 22402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:03.489818 22402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:03.490446 22402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:03.491437 22402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:03.491485 22402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.491554 22402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:03.491595 22402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.500528 22402 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:46733
I20260620 06:06:03.500555 22529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:46733 every 8 connection(s)
I20260620 06:06:03.501614 22402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:03.505748 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22402
I20260620 06:06:03.505889 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:03.506189 3532 external_mini_cluster.cc:1428] Setting key 716a46272241fb238f2c99d98c9f1545
I20260620 06:06:03.508423 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41303
--builtin_ntp_servers=127.3.115.20:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:06:03.512028 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.503504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:35973 (local address 127.3.115.62:41303)
0620 06:06:03.503653 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.503656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.504790 (+ 1134us) server_negotiation.cc:407] Connection header received
0620 06:06:03.506395 (+ 1605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.506399 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.506461 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.506561 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:03.507617 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.508369 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.509710 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.509966 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.510050 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.511426 (+ 1376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:03.511433 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:03.511439 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:03.511458 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:03.511486 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:03.511500 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:03.511585 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.511728 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.511805 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260620 06:06:03.512920 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.503933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:41303 (local address 127.3.115.2:35973)
0620 06:06:03.504602 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.504645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.506136 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.506783 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.506794 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.506923 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:03.507454 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.507468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.508508 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.508511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.509553 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.509567 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.510132 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.510137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.510379 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.511137 (+ 758us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:06:03.511139 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:03.511184 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:06:03.511260 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:03.511589 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:03.511595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:03.511599 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.511699 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.511943 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":118,"threads_started":1}
I20260620 06:06:03.514159 22530 heartbeater.cc:347] Connected to a master server at 127.3.115.62:41303
I20260620 06:06:03.514434 22530 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:03.514988 22530 heartbeater.cc:511] Master 127.3.115.62:41303 requested a full tablet report, sending...
I20260620 06:06:03.516047 22211 ts_manager.cc:194] Registered new tserver with Master: 0bd1457342fd479eb457c55e8a533ef8 (127.3.115.2:46733)
I20260620 06:06:03.516620 22211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.2:35973
W20260620 06:06:03.624218 22533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:03.624529 22533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:03.624605 22533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:03.628890 22533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:03.628971 22533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:03.629072 22533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:06:03.634249 22533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:41303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22533
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:03.635413 22533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:03.636880 22533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:03.647080 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.647081 22541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:03.647074 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:03.647454 22533 server_base.cc:1034] running on GCE node
I20260620 06:06:03.647876 22533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:03.648576 22533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:03.649778 22533 hybrid_clock.cc:648] HybridClock initialized: now 1781935563649760 us; error 37 us; skew 500 ppm
I20260620 06:06:03.651875 22533 webserver.cc:460] Webserver started at http://127.3.115.3:41721/ using document root <none> and password file <none>
I20260620 06:06:03.652525 22533 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:03.652624 22533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:03.652887 22533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:03.655225 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "41fc339b903d4fb2b9f37815cb8e52f1"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "3d14d35b05fda6acd667c240c1b2877e"
server_key_iv: "a55b0e6711cd8a28986d760f03b3208d"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.655864 22533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "41fc339b903d4fb2b9f37815cb8e52f1"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "3d14d35b05fda6acd667c240c1b2877e"
server_key_iv: "a55b0e6711cd8a28986d760f03b3208d"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.660066 22533 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260620 06:06:03.662840 22547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.664183 22533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:03.664368 22533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "41fc339b903d4fb2b9f37815cb8e52f1"
format_stamp: "Formatted at 2026-06-20 06:06:03 on dist-test-slave-4h14"
server_key: "3d14d35b05fda6acd667c240c1b2877e"
server_key_iv: "a55b0e6711cd8a28986d760f03b3208d"
server_key_version: "encryptionkey@0"
I20260620 06:06:03.664501 22533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:03.682682 22533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:03.683457 22533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:03.683727 22533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:03.684401 22533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:03.685570 22533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:03.685636 22533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.685703 22533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:03.685732 22533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:03.694612 22533 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:45929
I20260620 06:06:03.694631 22660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:45929 every 8 connection(s)
I20260620 06:06:03.695565 22533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:03.704839 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.697487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:59095 (local address 127.3.115.62:41303)
0620 06:06:03.697655 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.697660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.698776 (+ 1116us) server_negotiation.cc:407] Connection header received
0620 06:06:03.700161 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.700164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.700213 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.700298 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:03.701206 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.701751 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.702900 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.703119 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.703199 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.704322 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:03.704325 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:03.704327 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:03.704340 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:03.704360 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:03.704368 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:03.704427 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.704608 (+ 181us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.704708 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 06:06:03.705513 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22533
I20260620 06:06:03.705629 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:03.705967 3532 external_mini_cluster.cc:1428] Setting key 173ef9712fd78c86fc4de86aeb98ad54
I20260620 06:06:03.706081 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.697932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:41303 (local address 127.3.115.3:59095)
0620 06:06:03.698613 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.698655 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.699966 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.700467 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.700474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.700567 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:03.701077 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.701089 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.701873 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.701875 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.702790 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.702799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.703287 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.703290 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.703440 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.704032 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:06:03.704034 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:03.704074 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:06:03.704139 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:03.704467 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:03.704475 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:03.704481 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.704600 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.704932 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":117,"threads_started":1}
I20260620 06:06:03.707652 22661 heartbeater.cc:347] Connected to a master server at 127.3.115.62:41303
I20260620 06:06:03.707933 22661 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:03.708564 22661 heartbeater.cc:511] Master 127.3.115.62:41303 requested a full tablet report, sending...
I20260620 06:06:03.709740 22211 ts_manager.cc:194] Registered new tserver with Master: 41fc339b903d4fb2b9f37815cb8e52f1 (127.3.115.3:45929)
I20260620 06:06:03.710314 22211 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.3:59095
I20260620 06:06:03.720283 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:06:03.726001 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.722329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33024 (local address 127.3.115.62:41303)
0620 06:06:03.722445 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.722451 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.722587 (+ 136us) server_negotiation.cc:407] Connection header received
0620 06:06:03.722701 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.722704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.722765 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.722827 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:03.723089 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.723592 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.724398 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.724604 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.724734 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.725535 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:03.725539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:03.725541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:03.725552 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:03.725571 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:03.725578 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:03.725678 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.725850 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.725888 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:06:03.728955 22211 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:03.732040 22211 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:03.746402 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.740527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:46733 (local address 127.0.0.1:44216)
0620 06:06:03.740955 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.740999 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.741141 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.742024 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.742026 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.742038 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:03.742378 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.742383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.743925 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.743928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.744843 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.744857 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.745860 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.745864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.745966 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.746000 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.746111 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.746230 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":260,"spinlock_wait_cycles":3968,"thread_start_us":166,"threads_started":1}
I20260620 06:06:03.746403 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.741188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44216 (local address 127.3.115.2:46733)
0620 06:06:03.741608 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.741617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.741635 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:06:03.741697 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.741725 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.741887 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.742203 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:03.742603 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.743798 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.744965 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.745745 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.745861 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.746062 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.746150 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.746256 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":147,"threads_started":1}
I20260620 06:06:03.746899 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.741131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:45929 (local address 127.0.0.1:53258)
0620 06:06:03.741532 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.741545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.741636 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.742365 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.742368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.742384 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:03.742481 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.742487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.743486 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.743492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.745200 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.745207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.746265 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.746269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.746370 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.746391 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.746441 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.746739 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"mutex_wait_us":365,"thread_start_us":102,"threads_started":1}
I20260620 06:06:03.747133 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.740368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:44915 (local address 127.0.0.1:53108)
0620 06:06:03.740928 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.741000 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.741140 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.741969 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.741976 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.742005 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:03.742236 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.742244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.743906 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.743910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.745121 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.745132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.746265 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.746269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.746791 (+ 522us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.746825 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.746906 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.746980 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":525,"spinlock_wait_cycles":12032,"thread_start_us":71,"threads_started":1}
I20260620 06:06:03.747203 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.740507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.3.115.1:44915)
0620 06:06:03.741437 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.741450 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.741477 (+ 27us) server_negotiation.cc:407] Connection header received
0620 06:06:03.741560 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.741565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.741781 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.742510 (+ 729us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:03.742663 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.743777 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.745297 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.746115 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.746315 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.746667 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.746952 (+ 285us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.747057 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":823,"thread_start_us":175,"threads_started":1}
I20260620 06:06:03.747790 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.741470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53258 (local address 127.3.115.3:45929)
0620 06:06:03.741883 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.741897 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.741921 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:06:03.741997 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.742001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.742201 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.742388 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:03.742578 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.743370 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.745327 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.746115 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.747162 (+ 1047us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.747516 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.747565 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.747661 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":137,"threads_started":1}
I20260620 06:06:03.750231 22595 tablet_service.cc:1467] Processing CreateTablet for tablet 4a250b4de84644d0abd58eae5d8baaa2 (DEFAULT_TABLE table=test-table [id=6f0af341ef2e4cf2926690037d86235c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:03.750193 22464 tablet_service.cc:1467] Processing CreateTablet for tablet 4a250b4de84644d0abd58eae5d8baaa2 (DEFAULT_TABLE table=test-table [id=6f0af341ef2e4cf2926690037d86235c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:03.750193 22333 tablet_service.cc:1467] Processing CreateTablet for tablet 4a250b4de84644d0abd58eae5d8baaa2 (DEFAULT_TABLE table=test-table [id=6f0af341ef2e4cf2926690037d86235c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:03.751200 22595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a250b4de84644d0abd58eae5d8baaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:03.751219 22333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a250b4de84644d0abd58eae5d8baaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:03.751224 22464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a250b4de84644d0abd58eae5d8baaa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:03.758657 22678 tablet_bootstrap.cc:492] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Bootstrap starting.
I20260620 06:06:03.759111 22680 tablet_bootstrap.cc:492] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Bootstrap starting.
I20260620 06:06:03.761114 22679 tablet_bootstrap.cc:492] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: Bootstrap starting.
I20260620 06:06:03.761216 22680 tablet_bootstrap.cc:654] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:03.761216 22678 tablet_bootstrap.cc:654] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:03.762147 22680 log.cc:824] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:03.762147 22678 log.cc:824] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:03.763468 22679 tablet_bootstrap.cc:654] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:03.764191 22679 log.cc:824] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:03.764400 22680 tablet_bootstrap.cc:492] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: No bootstrap required, opened a new log
I20260620 06:06:03.764603 22680 ts_tablet_manager.cc:1397] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:03.765398 22678 tablet_bootstrap.cc:492] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: No bootstrap required, opened a new log
I20260620 06:06:03.765587 22678 ts_tablet_manager.cc:1397] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260620 06:06:03.766080 22679 tablet_bootstrap.cc:492] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: No bootstrap required, opened a new log
I20260620 06:06:03.766299 22679 ts_tablet_manager.cc:1397] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:06:03.769276 22680 raft_consensus.cc:348] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.769636 22680 raft_consensus.cc:374] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:03.769706 22680 raft_consensus.cc:729] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bd1457342fd479eb457c55e8a533ef8, State: Initialized, Role: FOLLOWER
I20260620 06:06:03.769528 22679 raft_consensus.cc:348] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.769783 22679 raft_consensus.cc:374] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:03.769845 22679 raft_consensus.cc:729] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8516eab892534d6280681fe31621d20a, State: Initialized, Role: FOLLOWER
I20260620 06:06:03.770184 22680 consensus_queue.cc:260] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.770300 22679 consensus_queue.cc:260] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.770912 22530 heartbeater.cc:503] Master 127.3.115.62:41303 was elected leader, sending a full tablet report...
I20260620 06:06:03.771292 22679 ts_tablet_manager.cc:1428] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260620 06:06:03.771292 22680 ts_tablet_manager.cc:1428] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260620 06:06:03.770138 22678 raft_consensus.cc:348] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.771598 22678 raft_consensus.cc:374] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:03.771626 22399 heartbeater.cc:503] Master 127.3.115.62:41303 was elected leader, sending a full tablet report...
I20260620 06:06:03.771667 22678 raft_consensus.cc:729] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41fc339b903d4fb2b9f37815cb8e52f1, State: Initialized, Role: FOLLOWER
I20260620 06:06:03.772149 22678 consensus_queue.cc:260] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.773130 22661 heartbeater.cc:503] Master 127.3.115.62:41303 was elected leader, sending a full tablet report...
I20260620 06:06:03.773198 22678 ts_tablet_manager.cc:1428] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260620 06:06:03.815456 22400 tablet.cc:2369] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:03.858759 22685 raft_consensus.cc:482] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:03.858968 22685 raft_consensus.cc:504] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.860146 22685 leader_election.cc:288] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0bd1457342fd479eb457c55e8a533ef8 (127.3.115.2:46733), 41fc339b903d4fb2b9f37815cb8e52f1 (127.3.115.3:45929)
I20260620 06:06:03.864374 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.860476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:33943 (local address 127.3.115.2:46733)
0620 06:06:03.860701 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.860708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.860770 (+ 62us) server_negotiation.cc:407] Connection header received
0620 06:06:03.860933 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.860935 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.860980 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.861055 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:03.861330 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.862001 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.863417 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.863808 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.863921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.863985 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.864145 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.864236 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260620 06:06:03.864387 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.860554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:44225 (local address 127.3.115.3:45929)
0620 06:06:03.860720 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.860725 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.860887 (+ 162us) server_negotiation.cc:407] Connection header received
0620 06:06:03.861038 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.861046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.861084 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.861163 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:03.861427 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.862090 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.863391 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.863816 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.863934 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.863994 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.864210 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.864282 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:03.864393 22688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.860449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:45929 (local address 127.3.115.1:44225)
0620 06:06:03.860805 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.860816 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.860930 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.861174 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.861177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.861192 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:03.861323 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.861328 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.862186 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.862190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.863269 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.863280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.863962 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.863966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.864068 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.864086 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.864179 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.864250 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":31,"thread_start_us":62,"threads_started":1}
I20260620 06:06:03.864392 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.860349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:46733 (local address 127.3.115.1:33943)
0620 06:06:03.860677 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.860693 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:03.860822 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:03.861078 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:03.861081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:03.861097 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:03.861219 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.861224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.862121 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.862127 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.863286 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:03.863295 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.863931 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.863934 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:03.864022 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.864045 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:06:03.864111 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 06:06:03.864207 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":94,"thread_start_us":100,"threads_started":1}
I20260620 06:06:03.865135 22615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a250b4de84644d0abd58eae5d8baaa2" candidate_uuid: "8516eab892534d6280681fe31621d20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" is_pre_election: true
I20260620 06:06:03.865190 22484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a250b4de84644d0abd58eae5d8baaa2" candidate_uuid: "8516eab892534d6280681fe31621d20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd1457342fd479eb457c55e8a533ef8" is_pre_election: true
I20260620 06:06:03.865486 22615 raft_consensus.cc:2454] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8516eab892534d6280681fe31621d20a in term 0.
I20260620 06:06:03.865486 22484 raft_consensus.cc:2454] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8516eab892534d6280681fe31621d20a in term 0.
I20260620 06:06:03.866014 22288 leader_election.cc:302] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0bd1457342fd479eb457c55e8a533ef8, 41fc339b903d4fb2b9f37815cb8e52f1, 8516eab892534d6280681fe31621d20a; no voters:
I20260620 06:06:03.866283 22685 raft_consensus.cc:2781] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:03.866376 22685 raft_consensus.cc:482] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:03.866427 22685 raft_consensus.cc:3037] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:03.867488 22685 raft_consensus.cc:504] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.867883 22685 leader_election.cc:288] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [CANDIDATE]: Term 1 election: Requested vote from peers 0bd1457342fd479eb457c55e8a533ef8 (127.3.115.2:46733), 41fc339b903d4fb2b9f37815cb8e52f1 (127.3.115.3:45929)
I20260620 06:06:03.868180 22484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a250b4de84644d0abd58eae5d8baaa2" candidate_uuid: "8516eab892534d6280681fe31621d20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bd1457342fd479eb457c55e8a533ef8"
I20260620 06:06:03.868415 22615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a250b4de84644d0abd58eae5d8baaa2" candidate_uuid: "8516eab892534d6280681fe31621d20a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41fc339b903d4fb2b9f37815cb8e52f1"
I20260620 06:06:03.868428 22484 raft_consensus.cc:3037] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:03.868542 22615 raft_consensus.cc:3037] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:03.869810 22484 raft_consensus.cc:2454] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8516eab892534d6280681fe31621d20a in term 1.
I20260620 06:06:03.869866 22615 raft_consensus.cc:2454] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8516eab892534d6280681fe31621d20a in term 1.
I20260620 06:06:03.870155 22288 leader_election.cc:302] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41fc339b903d4fb2b9f37815cb8e52f1, 8516eab892534d6280681fe31621d20a; no voters:
I20260620 06:06:03.870352 22685 raft_consensus.cc:2781] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:03.870574 22685 raft_consensus.cc:686] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [term 1 LEADER]: Becoming Leader. State: Replica: 8516eab892534d6280681fe31621d20a, State: Running, Role: LEADER
I20260620 06:06:03.870961 22685 consensus_queue.cc:237] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } }
I20260620 06:06:03.874269 22210 catalog_manager.cc:5495] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a reported cstate change: term changed from 0 to 1, leader changed from <none> to 8516eab892534d6280681fe31621d20a (127.3.115.1). New cstate: current_term: 1 leader_uuid: "8516eab892534d6280681fe31621d20a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8516eab892534d6280681fe31621d20a" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 44915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 } health_report { overall_health: UNKNOWN } } }
I20260620 06:06:03.900696 22674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.897269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.3.115.1:44915)
0620 06:06:03.897392 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:03.897396 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:03.897409 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:03.897502 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:03.897504 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:03.897562 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:03.897657 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:03.897877 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.898394 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.899304 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:03.899505 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:03.899578 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:03.899864 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:03.900281 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:03.900390 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:03.900509 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 06:06:03.900542 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:06:03.907292 22615 raft_consensus.cc:1261] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 1 FOLLOWER]: Refusing update from remote peer 8516eab892534d6280681fe31621d20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:03.907318 22484 raft_consensus.cc:1261] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 1 FOLLOWER]: Refusing update from remote peer 8516eab892534d6280681fe31621d20a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:03.908042 22685 consensus_queue.cc:1035] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bd1457342fd479eb457c55e8a533ef8" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 46733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:03.908245 22689 consensus_queue.cc:1035] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a [LEADER]: Connected to new peer: Peer: permanent_uuid: "41fc339b903d4fb2b9f37815cb8e52f1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 45929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:03.917619 22693 mvcc.cc:205] Tried to move back new op lower bound from 7298808069756895232 to 7298808069618438144. Current Snapshot: MvccSnapshot[applied={T|T < 7298808069756895232}]
I20260620 06:06:03.918454 22694 mvcc.cc:205] Tried to move back new op lower bound from 7298808069756895232 to 7298808069618438144. Current Snapshot: MvccSnapshot[applied={T|T < 7298808069756895232}]
I20260620 06:06:03.926738 22210 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:03.926919 22210 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33024:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:03.929345 22210 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5d69f3578cfc42e197f61f324b1148b8: Sending DeleteTablet for 3 replicas of tablet 4a250b4de84644d0abd58eae5d8baaa2
I20260620 06:06:03.930117 22333 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a250b4de84644d0abd58eae5d8baaa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:03 UTC) from {username='slave'} at 127.0.0.1:53108
I20260620 06:06:03.930147 22595 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a250b4de84644d0abd58eae5d8baaa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:03 UTC) from {username='slave'} at 127.0.0.1:53258
I20260620 06:06:03.930128 22464 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a250b4de84644d0abd58eae5d8baaa2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:03 UTC) from {username='slave'} at 127.0.0.1:44216
I20260620 06:06:03.930610 22702 tablet_replica.cc:330] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: stopping tablet replica
I20260620 06:06:03.930830 22703 tablet_replica.cc:330] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: stopping tablet replica
I20260620 06:06:03.930657 22704 tablet_replica.cc:330] T 4a250b4de84644d0abd58eae5d8baaa2 P 8516eab892534d6280681fe31621d20a: stopping tablet replica
I20260620 06:06:03.930950 22702 raft_consensus.cc:2229] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:03.931334 22702 raft_consensus.cc:2258] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:03.931367 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22271
I20260620 06:06:03.932830 22703 raft_consensus.cc:2229] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:03.933151 22703 raft_consensus.cc:2258] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:03.934057 22702 ts_tablet_manager.cc:1905] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:03.936235 22702 ts_tablet_manager.cc:1918] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:03.936318 22702 log.cc:1196] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/4a250b4de84644d0abd58eae5d8baaa2
I20260620 06:06:03.936600 22702 ts_tablet_manager.cc:1939] T 4a250b4de84644d0abd58eae5d8baaa2 P 0bd1457342fd479eb457c55e8a533ef8: Deleting consensus metadata
I20260620 06:06:03.937686 22703 ts_tablet_manager.cc:1905] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:03.937773 22196 catalog_manager.cc:4841] TS 0bd1457342fd479eb457c55e8a533ef8 (127.3.115.2:46733): tablet 4a250b4de84644d0abd58eae5d8baaa2 (table test-table [id=6f0af341ef2e4cf2926690037d86235c]) successfully deleted
W20260620 06:06:03.939466 22198 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:44915 (error 108)
I20260620 06:06:03.940378 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22402
I20260620 06:06:03.940514 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:03.940217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:44915 (local address 127.0.0.1:53138)
0620 06:06:03.940353 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:03.940423 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:44915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 06:06:03.940935 22703 ts_tablet_manager.cc:1918] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:03.941040 22703 log.cc:1196] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/4a250b4de84644d0abd58eae5d8baaa2
I20260620 06:06:03.941372 22703 ts_tablet_manager.cc:1939] T 4a250b4de84644d0abd58eae5d8baaa2 P 41fc339b903d4fb2b9f37815cb8e52f1: Deleting consensus metadata
W20260620 06:06:03.941663 22198 catalog_manager.cc:4584] TS 8516eab892534d6280681fe31621d20a (127.3.115.1:44915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a250b4de84644d0abd58eae5d8baaa2: Network error: Client connection negotiation failed: client connection to 127.3.115.1:44915: connect: Connection refused (error 111)
I20260620 06:06:03.942754 22197 catalog_manager.cc:4841] TS 41fc339b903d4fb2b9f37815cb8e52f1 (127.3.115.3:45929): tablet 4a250b4de84644d0abd58eae5d8baaa2 (table test-table [id=6f0af341ef2e4cf2926690037d86235c]) successfully deleted
I20260620 06:06:03.946579 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22533
I20260620 06:06:03.952301 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22180
2026-06-20T06:06:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1106 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:04 dist-test-slave-4h14 krb5kdc[22708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:04 dist-test-slave-4h14 krb5kdc[22708](info): set up 2 sockets
Jun 20 06:06:04 dist-test-slave-4h14 krb5kdc[22708](info): commencing operation
krb5kdc: starting...
W20260620 06:06:04.996860 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:05.126039 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35125
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:35125
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:05.243132 22721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:05.243427 22721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:05.243470 22721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:05.247712 22721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:05.247794 22721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:05.247814 22721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:05.247833 22721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:05.247892 22721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:05.253645 22721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:35125
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:35125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:05.254823 22721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:05.255913 22721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:05.265156 22727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:05.265159 22726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:05.265158 22729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:05.265618 22721 server_base.cc:1034] running on GCE node
I20260620 06:06:05.266251 22721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:05.267490 22721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:05.268754 22721 hybrid_clock.cc:648] HybridClock initialized: now 1781935565268710 us; error 38 us; skew 500 ppm
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:05.271812 22721 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:05.272883 22721 webserver.cc:460] Webserver started at http://127.3.115.62:43619/ using document root <none> and password file <none>
I20260620 06:06:05.273461 22721 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:05.273521 22721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:05.273694 22721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:05.275796 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "d6ead750e11a420c85e2a5a93742fc65"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "5f0bbd32b4632b6333fa5b705ebfeca5"
server_key_iv: "8028c04698b5241a0093ff6d5eabe2a2"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.276332 22721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "d6ead750e11a420c85e2a5a93742fc65"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "5f0bbd32b4632b6333fa5b705ebfeca5"
server_key_iv: "8028c04698b5241a0093ff6d5eabe2a2"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.280524 22721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:05.284257 22736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.286332 22721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260620 06:06:05.286481 22721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "d6ead750e11a420c85e2a5a93742fc65"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "5f0bbd32b4632b6333fa5b705ebfeca5"
server_key_iv: "8028c04698b5241a0093ff6d5eabe2a2"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.286629 22721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:05.312075 22721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:05.315408 22721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:05.315574 22721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:05.323323 22721 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:35125
I20260620 06:06:05.323354 22788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:35125 every 8 connection(s)
I20260620 06:06:05.324365 22721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:05.327324 22789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:05.333355 22789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65: Bootstrap starting.
I20260620 06:06:05.333408 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22721
I20260620 06:06:05.333534 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:05.333846 3532 external_mini_cluster.cc:1428] Setting key 752197189e49014919d0715a7495c68f
I20260620 06:06:05.336215 22789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:05.337046 22789 log.cc:824] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:05.339306 22789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65: No bootstrap required, opened a new log
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:05.344098 22789 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ead750e11a420c85e2a5a93742fc65" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35125 } }
I20260620 06:06:05.344383 22789 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:05.344461 22789 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ead750e11a420c85e2a5a93742fc65, State: Initialized, Role: FOLLOWER
I20260620 06:06:05.345016 22789 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ead750e11a420c85e2a5a93742fc65" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35125 } }
I20260620 06:06:05.345182 22789 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:05.345276 22789 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:05.345399 22789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:05.346378 22789 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ead750e11a420c85e2a5a93742fc65" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35125 } }
I20260620 06:06:05.346750 22789 leader_election.cc:302] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6ead750e11a420c85e2a5a93742fc65; no voters:
I20260620 06:06:05.347082 22789 leader_election.cc:288] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:05.347219 22794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:05.347461 22794 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [term 1 LEADER]: Becoming Leader. State: Replica: d6ead750e11a420c85e2a5a93742fc65, State: Running, Role: LEADER
I20260620 06:06:05.347795 22794 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ead750e11a420c85e2a5a93742fc65" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35125 } }
I20260620 06:06:05.348584 22789 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:05.349524 22795 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6ead750e11a420c85e2a5a93742fc65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ead750e11a420c85e2a5a93742fc65" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35125 } } }
I20260620 06:06:05.349676 22795 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:05.350203 22796 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ead750e11a420c85e2a5a93742fc65. Latest consensus state: current_term: 1 leader_uuid: "d6ead750e11a420c85e2a5a93742fc65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6ead750e11a420c85e2a5a93742fc65" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 35125 } } }
I20260620 06:06:05.350283 22796 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:05.350087 22803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:05.350909 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:05.335336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.3.115.62:35125)
0620 06:06:05.335930 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:05.335947 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:05.335985 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:06:05.336868 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:05.336889 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:05.337290 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:05.337663 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:05.338522 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.339607 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.340766 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.341247 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.341376 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:05.344380 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:05.344401 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:05.344424 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:05.344462 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:05.346613 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:05.347209 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:05.347215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:05.347222 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:05.347392 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:05.347718 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:05.347721 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:05.347723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:05.348266 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:05.348472 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:05.348826 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 06:06:05.349177 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":138,"threads_started":1}
I20260620 06:06:05.353854 22803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:05.359844 22803 catalog_manager.cc:1347] Generated new cluster ID: 5df5ebac51d84bfc9a7a4fefb041cd45
I20260620 06:06:05.359931 22803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:05.383095 22803 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:05.384138 22803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:05.390794 22803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65: Generated new TSK 0
I20260620 06:06:05.391571 22803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.115.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
Entry for principal kudu/127.3.115.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab.
I20260620 06:06:05.423211 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35125
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:05.540370 22815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:05.540756 22815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:05.540868 22815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:05.545298 22815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:05.545408 22815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:05.545501 22815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:06:05.551505 22815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.1.keytab
--principal=kudu/127.3.115.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:05.552928 22815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:05.554075 22815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:05.564445 22820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:05.564419 22823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:05.564465 22821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:05.564684 22815 server_base.cc:1034] running on GCE node
I20260620 06:06:05.565172 22815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:05.565866 22815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:05.567046 22815 hybrid_clock.cc:648] HybridClock initialized: now 1781935565567027 us; error 34 us; skew 500 ppm
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:05.570061 22815 init.cc:377] Logged in from keytab as kudu/127.3.115.1@KRBTEST.COM (short username kudu)
I20260620 06:06:05.570992 22815 webserver.cc:460] Webserver started at http://127.3.115.1:42743/ using document root <none> and password file <none>
I20260620 06:06:05.571548 22815 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:05.571610 22815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:05.571779 22815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:05.573917 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "90d18fee8d86490cb8a7c520c736697b"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "fa842f2ae269cae1250cc7abf2931f06"
server_key_iv: "be8ac9703fcafd201bc840eee3bdad6a"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.574443 22815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "90d18fee8d86490cb8a7c520c736697b"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "fa842f2ae269cae1250cc7abf2931f06"
server_key_iv: "be8ac9703fcafd201bc840eee3bdad6a"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.578508 22815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:05.581121 22830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.582517 22815 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:05.582631 22815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "90d18fee8d86490cb8a7c520c736697b"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "fa842f2ae269cae1250cc7abf2931f06"
server_key_iv: "be8ac9703fcafd201bc840eee3bdad6a"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.582732 22815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:05.607758 22815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:05.611346 22815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:05.611517 22815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:05.612093 22815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:05.612962 22815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:05.613024 22815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.613067 22815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:05.613088 22815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.622298 22815 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:35187
I20260620 06:06:05.622319 22943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:35187 every 8 connection(s)
I20260620 06:06:05.623287 22815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:05.630707 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22815
I20260620 06:06:05.630884 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:05.631184 3532 external_mini_cluster.cc:1428] Setting key d0ae0500c843e0cb0f26ed81d8b9352c
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.1@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:05.638084 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:05.625217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:35179 (local address 127.3.115.62:35125)
0620 06:06:05.625367 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:05.625370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:05.626283 (+ 913us) server_negotiation.cc:407] Connection header received
0620 06:06:05.627676 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:05.627680 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:05.627740 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:05.627843 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:05.629122 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.629662 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.630848 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.631152 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.631262 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:05.634310 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:05.634315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:05.634317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:05.634351 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:05.635835 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:05.636390 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:05.636394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:05.636396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:05.636477 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:05.636921 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:05.636924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:05.636926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:05.637129 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:05.637234 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:05.637769 (+ 535us) server_negotiation.cc:299] Negotiation successful
0620 06:06:05.637908 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 06:06:05.639097 22946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:05.625588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35125 (local address 127.3.115.1:35179)
0620 06:06:05.626112 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:05.626160 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:05.627445 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:05.628035 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:05.628043 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:05.628418 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:05.628954 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:05.628969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.629798 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.629806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:05.630717 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:05.630726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.631312 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.631315 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:05.631495 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:05.632307 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:05.632328 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:05.634129 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:05.635964 (+ 1835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:05.635971 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:05.636001 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:05.636280 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:05.636575 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:05.636579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:05.636581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:05.636816 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:05.637234 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:05.637242 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:05.637525 (+ 283us) client_negotiation.cc:769] Sending connection context
0620 06:06:05.637718 (+ 193us) client_negotiation.cc:240] Negotiation successful
0620 06:06:05.638111 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":130,"threads_started":1}
WARNING: no policy specified for kudu/127.3.115.2@KRBTEST.COM; defaulting to no policy
I20260620 06:06:05.640722 22944 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35125
I20260620 06:06:05.641000 22944 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:05.641603 22944 heartbeater.cc:511] Master 127.3.115.62:35125 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.2@KRBTEST.COM" created.
I20260620 06:06:05.643476 22753 ts_manager.cc:194] Registered new tserver with Master: 90d18fee8d86490cb8a7c520c736697b (127.3.115.1:35187)
I20260620 06:06:05.645252 22753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.1@KRBTEST.COM'} at 127.3.115.1:35179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
Entry for principal kudu/127.3.115.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab.
I20260620 06:06:05.659801 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35125
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:05.775254 22949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:05.775563 22949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:05.775609 22949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:05.779718 22949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:05.779798 22949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:05.779860 22949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:06:05.785267 22949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.2.keytab
--principal=kudu/127.3.115.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:05.786882 22949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:05.788048 22949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:05.798013 22957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:05.798015 22954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:05.798013 22955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:05.798358 22949 server_base.cc:1034] running on GCE node
I20260620 06:06:05.799088 22949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:05.799768 22949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:05.800943 22949 hybrid_clock.cc:648] HybridClock initialized: now 1781935565800922 us; error 37 us; skew 500 ppm
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:05.803776 22949 init.cc:377] Logged in from keytab as kudu/127.3.115.2@KRBTEST.COM (short username kudu)
I20260620 06:06:05.804775 22949 webserver.cc:460] Webserver started at http://127.3.115.2:41165/ using document root <none> and password file <none>
I20260620 06:06:05.805351 22949 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:05.805442 22949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:05.805689 22949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:05.807756 22949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "f0889709de474fc68bf6ac445c3e9d8d"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "a6f74628d504e71e30a034eb9bb5cb84"
server_key_iv: "150c44f3f7f2874891ae48b2963ed423"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.808321 22949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "f0889709de474fc68bf6ac445c3e9d8d"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "a6f74628d504e71e30a034eb9bb5cb84"
server_key_iv: "150c44f3f7f2874891ae48b2963ed423"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.812074 22949 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260620 06:06:05.814710 22964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.816056 22949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:05.816205 22949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "f0889709de474fc68bf6ac445c3e9d8d"
format_stamp: "Formatted at 2026-06-20 06:06:05 on dist-test-slave-4h14"
server_key: "a6f74628d504e71e30a034eb9bb5cb84"
server_key_iv: "150c44f3f7f2874891ae48b2963ed423"
server_key_version: "encryptionkey@0"
I20260620 06:06:05.816359 22949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:05.839583 22949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:05.843245 22949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:05.843470 22949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:05.844216 22949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:05.845283 22949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:05.845355 22949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.845423 22949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:05.845466 22949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:05.855178 22949 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:42163
I20260620 06:06:05.855199 23077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:42163 every 8 connection(s)
I20260620 06:06:05.856146 22949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:05.856832 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 22949
I20260620 06:06:05.856974 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:05.857317 3532 external_mini_cluster.cc:1428] Setting key 8cdd6c02ff2ecd341a8a1ec1b19fe1ae
Jun 20 06:06:05 dist-test-slave-4h14 krb5kdc[22708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935565, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.2@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.115.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.3@KRBTEST.COM" created.
I20260620 06:06:05.873344 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:05.858268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:54587 (local address 127.3.115.62:35125)
0620 06:06:05.858398 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:05.858401 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:05.859592 (+ 1191us) server_negotiation.cc:407] Connection header received
0620 06:06:05.861291 (+ 1699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:05.861294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:05.861341 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:05.861417 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:05.862718 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.863246 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.865252 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.865471 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.865552 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:05.868877 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:05.868881 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:05.868882 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:05.868910 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:05.870424 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:05.871105 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:05.871108 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:05.871110 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:05.871197 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:05.871680 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:05.871694 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:05.871699 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:05.871990 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:05.872114 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:05.872904 (+ 790us) server_negotiation.cc:299] Negotiation successful
0620 06:06:05.873071 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:06:05.874406 23080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:05.858853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35125 (local address 127.3.115.2:54587)
0620 06:06:05.859425 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:05.859501 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:05.861007 (+ 1506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:05.861632 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:05.861642 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:05.862070 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:05.862579 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:05.862594 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.863388 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.863395 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:05.865082 (+ 1687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:05.865100 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:05.865643 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:05.865648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:05.865915 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:05.866638 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:05.866664 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:05.868600 (+ 1936us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:05.870551 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:05.870557 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:05.870575 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:05.870978 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:05.871318 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:05.871321 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:05.871323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:05.871552 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:05.872127 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:05.872136 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:05.872546 (+ 410us) client_negotiation.cc:769] Sending connection context
0620 06:06:05.872812 (+ 266us) client_negotiation.cc:240] Negotiation successful
0620 06:06:05.873307 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":151,"threads_started":1}
I20260620 06:06:05.875694 23078 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35125
I20260620 06:06:05.875998 23078 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:05.876701 23078 heartbeater.cc:511] Master 127.3.115.62:35125 requested a full tablet report, sending...
I20260620 06:06:05.877928 22753 ts_manager.cc:194] Registered new tserver with Master: f0889709de474fc68bf6ac445c3e9d8d (127.3.115.2:42163)
I20260620 06:06:05.878517 22753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.2@KRBTEST.COM'} at 127.3.115.2:54587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
Entry for principal kudu/127.3.115.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab.
I20260620 06:06:05.889928 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35125
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:06.002655 23083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:06.003002 23083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:06.003078 23083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:06.007139 23083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:06.007239 23083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:06.007329 23083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:06:06.013489 23083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.3.keytab
--principal=kudu/127.3.115.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:35125
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:06.014667 23083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:06.015738 23083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:06.025753 23089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:06.025858 23083 server_base.cc:1034] running on GCE node
W20260620 06:06:06.025887 23091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:06.026033 23088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:06.026567 23083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:06.027215 23083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:06.028375 23083 hybrid_clock.cc:648] HybridClock initialized: now 1781935566028359 us; error 35 us; skew 500 ppm
Jun 20 06:06:06 dist-test-slave-4h14 krb5kdc[22708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:06.031340 23083 init.cc:377] Logged in from keytab as kudu/127.3.115.3@KRBTEST.COM (short username kudu)
I20260620 06:06:06.032256 23083 webserver.cc:460] Webserver started at http://127.3.115.3:36885/ using document root <none> and password file <none>
I20260620 06:06:06.032883 23083 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:06.032950 23083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:06.033123 23083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:06.035187 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "3175b569f4f7467f92f0893d6729efb1"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "5ba6a2fd81e61682dd4575c647d59187"
server_key_iv: "9db81388e2f4cc92cc00d39d2d098fae"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.035704 23083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "3175b569f4f7467f92f0893d6729efb1"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "5ba6a2fd81e61682dd4575c647d59187"
server_key_iv: "9db81388e2f4cc92cc00d39d2d098fae"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.039618 23083 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:06:06.042222 23098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.043619 23083 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:06.043766 23083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "3175b569f4f7467f92f0893d6729efb1"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "5ba6a2fd81e61682dd4575c647d59187"
server_key_iv: "9db81388e2f4cc92cc00d39d2d098fae"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.043881 23083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:06.059437 23083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:06.062722 23083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:06.062923 23083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:06.063544 23083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:06.064392 23083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:06.064441 23083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.064527 23083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:06.064545 23083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.073810 23083 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:39321
I20260620 06:06:06.073884 23211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:39321 every 8 connection(s)
I20260620 06:06:06.074790 23083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:06.076803 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23083
I20260620 06:06:06.076961 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:06.077263 3532 external_mini_cluster.cc:1428] Setting key 718c88d7abcc3ca8f76f5fec6dffbbad
Jun 20 06:06:06 dist-test-slave-4h14 krb5kdc[22708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935566, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.3@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:06.089596 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.076691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:40515 (local address 127.3.115.62:35125)
0620 06:06:06.077705 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.077710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.077948 (+ 238us) server_negotiation.cc:407] Connection header received
0620 06:06:06.079543 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.079545 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.079591 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.079648 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:06.081094 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.081653 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.082823 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.083045 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.083128 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.085771 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:06.085774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:06.085776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:06.085802 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:06.087336 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:06.087937 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:06.087940 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:06.087941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:06.088055 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:06.088461 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:06.088464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:06.088466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:06.088711 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:06.088792 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.089346 (+ 554us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.089462 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920}
I20260620 06:06:06.090557 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.077189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:35125 (local address 127.3.115.3:40515)
0620 06:06:06.077746 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.077821 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.079334 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.079796 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.079804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.080250 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:06.080926 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.080944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.081773 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.081775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.082705 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.082714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.083189 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.083192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.083374 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.083921 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:06.083938 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:06.085570 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:06.087463 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:06.087475 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:06.087494 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:06.087818 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:06.088141 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:06.088143 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:06.088144 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:06.088302 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:06.088822 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:06.088829 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:06.089094 (+ 265us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.089262 (+ 168us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.089562 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":155,"threads_started":1}
I20260620 06:06:06.091988 23212 heartbeater.cc:347] Connected to a master server at 127.3.115.62:35125
I20260620 06:06:06.092255 23212 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:06.092829 23212 heartbeater.cc:511] Master 127.3.115.62:35125 requested a full tablet report, sending...
I20260620 06:06:06.093891 22753 ts_manager.cc:194] Registered new tserver with Master: 3175b569f4f7467f92f0893d6729efb1 (127.3.115.3:39321)
I20260620 06:06:06.094489 22753 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.115.3@KRBTEST.COM'} at 127.3.115.3:40515
I20260620 06:06:06.104372 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:06:06.113946 22792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.106411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50740 (local address 127.3.115.62:35125)
0620 06:06:06.106565 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.106570 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.106732 (+ 162us) server_negotiation.cc:407] Connection header received
0620 06:06:06.106882 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.106885 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.106932 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.107015 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:06.107752 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.108285 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.109221 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.109431 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.109536 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.110901 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:06.110905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:06.110908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:06.110937 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:06.112213 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:06.112632 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:06.112636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:06.112638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:06.112756 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:06.113020 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:06.113024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:06.113026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:06.113272 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:06.113413 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.113643 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.113774 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:06.116751 22753 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:06.119772 22753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:06.134977 23224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.128517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:39321 (local address 127.0.0.1:33250)
0620 06:06:06.129706 (+ 1189us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.129720 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.129815 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.130301 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.130304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.130316 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:06.130973 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.130978 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.131947 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.131950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.133583 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.133592 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.134424 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.134428 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.134530 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.134565 (+ 35us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.134691 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.134815 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"mutex_wait_us":678,"thread_start_us":160,"threads_started":1}
I20260620 06:06:06.134984 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.128946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33250 (local address 127.3.115.3:39321)
0620 06:06:06.129600 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.129609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.129901 (+ 292us) server_negotiation.cc:407] Connection header received
0620 06:06:06.129973 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.129979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.130162 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.130309 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:06.131075 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.131809 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.133733 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.134297 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.134423 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.134578 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.134720 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.134820 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":120,"threads_started":1}
I20260620 06:06:06.135473 23227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.129251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42163 (local address 127.0.0.1:51290)
0620 06:06:06.130189 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.130200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.130309 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.130722 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.130725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.130742 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:06.130839 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.130846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.132225 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.132228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.133581 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.133592 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.134727 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.134731 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.134968 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.135005 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.135260 (+ 255us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.135326 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"mutex_wait_us":220,"thread_start_us":79,"threads_started":1}
I20260620 06:06:06.135896 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.129463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51290 (local address 127.3.115.2:42163)
0620 06:06:06.130080 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.130092 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.130371 (+ 279us) server_negotiation.cc:407] Connection header received
0620 06:06:06.130447 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.130451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.130589 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.130741 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:06.130984 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.132075 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.133706 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.134476 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.134568 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.134678 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.135643 (+ 965us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.135749 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":144,"threads_started":1}
I20260620 06:06:06.135933 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.128594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41026 (local address 127.3.115.1:35187)
0620 06:06:06.129435 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.129451 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.129481 (+ 30us) server_negotiation.cc:407] Connection header received
0620 06:06:06.129732 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.129741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.129971 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.130525 (+ 554us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:06.130662 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.131740 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.133518 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.134356 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.134469 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.134625 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.135543 (+ 918us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.135784 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":156,"threads_started":1}
I20260620 06:06:06.135474 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.128476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35187 (local address 127.0.0.1:41026)
0620 06:06:06.129360 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.129413 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.129542 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.130136 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.130143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.130179 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:06.130370 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.130379 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.131893 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.131898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.133394 (+ 1496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.133403 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.134690 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.134697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.134941 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.134989 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.135062 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.135351 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":327,"thread_start_us":86,"threads_started":1}
I20260620 06:06:06.138022 23146 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6e1eccddc74f3a899337acc504d0a2 (DEFAULT_TABLE table=test-table [id=4c13f07e79d1439ba96e3062f79fa1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:06.138466 22878 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6e1eccddc74f3a899337acc504d0a2 (DEFAULT_TABLE table=test-table [id=4c13f07e79d1439ba96e3062f79fa1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:06.139024 23146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6e1eccddc74f3a899337acc504d0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:06.139411 22878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6e1eccddc74f3a899337acc504d0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:06.145452 23230 tablet_bootstrap.cc:492] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Bootstrap starting.
I20260620 06:06:06.146320 23012 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6e1eccddc74f3a899337acc504d0a2 (DEFAULT_TABLE table=test-table [id=4c13f07e79d1439ba96e3062f79fa1c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:06.147241 23012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6e1eccddc74f3a899337acc504d0a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:06.147800 23230 tablet_bootstrap.cc:654] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:06.149050 23230 log.cc:824] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:06.149147 23229 tablet_bootstrap.cc:492] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: Bootstrap starting.
I20260620 06:06:06.151322 23230 tablet_bootstrap.cc:492] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: No bootstrap required, opened a new log
I20260620 06:06:06.151460 23229 tablet_bootstrap.cc:654] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:06.151535 23230 ts_tablet_manager.cc:1397] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260620 06:06:06.152328 23229 log.cc:824] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:06.154521 23229 tablet_bootstrap.cc:492] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: No bootstrap required, opened a new log
I20260620 06:06:06.154502 23233 tablet_bootstrap.cc:492] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Bootstrap starting.
I20260620 06:06:06.154736 23229 ts_tablet_manager.cc:1397] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20260620 06:06:06.155514 23230 raft_consensus.cc:348] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.155833 23230 raft_consensus.cc:374] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:06.155903 23230 raft_consensus.cc:729] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3175b569f4f7467f92f0893d6729efb1, State: Initialized, Role: FOLLOWER
I20260620 06:06:06.156430 23230 consensus_queue.cc:260] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.157191 23212 heartbeater.cc:503] Master 127.3.115.62:35125 was elected leader, sending a full tablet report...
I20260620 06:06:06.157557 23230 ts_tablet_manager.cc:1428] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:06:06.158072 23233 tablet_bootstrap.cc:654] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:06.159025 23233 log.cc:824] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:06.159416 23229 raft_consensus.cc:348] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.159650 23229 raft_consensus.cc:374] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:06.159732 23229 raft_consensus.cc:729] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d18fee8d86490cb8a7c520c736697b, State: Initialized, Role: FOLLOWER
I20260620 06:06:06.160262 23229 consensus_queue.cc:260] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.161263 22944 heartbeater.cc:503] Master 127.3.115.62:35125 was elected leader, sending a full tablet report...
I20260620 06:06:06.161561 23229 ts_tablet_manager.cc:1428] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260620 06:06:06.161594 23233 tablet_bootstrap.cc:492] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: No bootstrap required, opened a new log
I20260620 06:06:06.161824 23233 ts_tablet_manager.cc:1397] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20260620 06:06:06.165731 23233 raft_consensus.cc:348] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.166042 23233 raft_consensus.cc:374] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:06.166116 23233 raft_consensus.cc:729] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0889709de474fc68bf6ac445c3e9d8d, State: Initialized, Role: FOLLOWER
I20260620 06:06:06.166599 23233 consensus_queue.cc:260] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.167264 23078 heartbeater.cc:503] Master 127.3.115.62:35125 was elected leader, sending a full tablet report...
I20260620 06:06:06.167544 23233 ts_tablet_manager.cc:1428] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:06:06.322921 23234 raft_consensus.cc:482] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:06.323179 23234 raft_consensus.cc:504] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.324409 23234 leader_election.cc:288] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90d18fee8d86490cb8a7c520c736697b (127.3.115.1:35187), f0889709de474fc68bf6ac445c3e9d8d (127.3.115.2:42163)
W20260620 06:06:06.325724 23213 tablet.cc:2369] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:06.329075 23228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.325297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:33435 (local address 127.3.115.2:42163)
0620 06:06:06.325441 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.325447 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.325463 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:06:06.325508 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.325511 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.325562 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.325634 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:06.326106 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.326799 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.327899 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.328438 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.328555 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.328620 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.328867 (+ 247us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.328946 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:06.329087 23238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.324727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42163 (local address 127.3.115.3:33435)
0620 06:06:06.325259 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.325273 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.325361 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.325750 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.325752 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.325767 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:06.325971 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.325979 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.326906 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.326910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.327771 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.327780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.328570 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.328574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.328725 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.328751 (+ 26us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.328841 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.328916 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":124,"threads_started":1}
I20260620 06:06:06.329466 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.324615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35187 (local address 127.3.115.3:54237)
0620 06:06:06.324773 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.324792 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.324894 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.325488 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.325491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.325505 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:06.325634 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.325641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.326693 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.326696 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.327975 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.327985 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.328995 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.328999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.329117 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.329142 (+ 25us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.329224 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.329318 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":52}
I20260620 06:06:06.329744 23225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.324853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:54237 (local address 127.3.115.1:35187)
0620 06:06:06.325204 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.325210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.325228 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:06:06.325284 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.325287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.325334 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.325417 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:06.325766 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.326570 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.328133 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.328767 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.328879 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.328938 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.329545 (+ 607us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.329630 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262}
I20260620 06:06:06.329898 23032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6e1eccddc74f3a899337acc504d0a2" candidate_uuid: "3175b569f4f7467f92f0893d6729efb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0889709de474fc68bf6ac445c3e9d8d" is_pre_election: true
I20260620 06:06:06.330251 23032 raft_consensus.cc:2454] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3175b569f4f7467f92f0893d6729efb1 in term 0.
I20260620 06:06:06.330289 22898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6e1eccddc74f3a899337acc504d0a2" candidate_uuid: "3175b569f4f7467f92f0893d6729efb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d18fee8d86490cb8a7c520c736697b" is_pre_election: true
I20260620 06:06:06.330572 22898 raft_consensus.cc:2454] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3175b569f4f7467f92f0893d6729efb1 in term 0.
I20260620 06:06:06.330783 23102 leader_election.cc:302] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3175b569f4f7467f92f0893d6729efb1, f0889709de474fc68bf6ac445c3e9d8d; no voters:
I20260620 06:06:06.331070 23234 raft_consensus.cc:2781] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:06.331169 23234 raft_consensus.cc:482] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:06.331223 23234 raft_consensus.cc:3037] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:06.332262 23234 raft_consensus.cc:504] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.332644 23234 leader_election.cc:288] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [CANDIDATE]: Term 1 election: Requested vote from peers 90d18fee8d86490cb8a7c520c736697b (127.3.115.1:35187), f0889709de474fc68bf6ac445c3e9d8d (127.3.115.2:42163)
I20260620 06:06:06.333031 22898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6e1eccddc74f3a899337acc504d0a2" candidate_uuid: "3175b569f4f7467f92f0893d6729efb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d18fee8d86490cb8a7c520c736697b"
I20260620 06:06:06.333166 22898 raft_consensus.cc:3037] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:06.334223 22898 raft_consensus.cc:2454] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3175b569f4f7467f92f0893d6729efb1 in term 1.
I20260620 06:06:06.334546 23032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e6e1eccddc74f3a899337acc504d0a2" candidate_uuid: "3175b569f4f7467f92f0893d6729efb1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0889709de474fc68bf6ac445c3e9d8d"
I20260620 06:06:06.334571 23101 leader_election.cc:302] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3175b569f4f7467f92f0893d6729efb1, 90d18fee8d86490cb8a7c520c736697b; no voters:
I20260620 06:06:06.334759 23234 raft_consensus.cc:2781] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:06.334689 23032 raft_consensus.cc:3037] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:06.335000 23234 raft_consensus.cc:686] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 1 LEADER]: Becoming Leader. State: Replica: 3175b569f4f7467f92f0893d6729efb1, State: Running, Role: LEADER
I20260620 06:06:06.335373 23234 consensus_queue.cc:237] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } }
I20260620 06:06:06.336074 23032 raft_consensus.cc:2454] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3175b569f4f7467f92f0893d6729efb1 in term 1.
I20260620 06:06:06.338959 22753 catalog_manager.cc:5495] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3175b569f4f7467f92f0893d6729efb1 (127.3.115.3). New cstate: current_term: 1 leader_uuid: "3175b569f4f7467f92f0893d6729efb1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3175b569f4f7467f92f0893d6729efb1" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 } health_report { overall_health: UNKNOWN } } }
W20260620 06:06:06.357502 23079 tablet.cc:2369] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:06.370018 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.366658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33252 (local address 127.3.115.3:39321)
0620 06:06:06.366798 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.366802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.366816 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:06:06.366923 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.366925 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.366989 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.367082 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:06.367291 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.367804 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.368593 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.368872 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.368949 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.369222 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:06.369660 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:06.369780 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.369827 (+ 47us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.369883 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260620 06:06:06.374583 22945 tablet.cc:2369] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:06.376353 23032 raft_consensus.cc:1261] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 1 FOLLOWER]: Refusing update from remote peer 3175b569f4f7467f92f0893d6729efb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:06.376353 22898 raft_consensus.cc:1261] T 1e6e1eccddc74f3a899337acc504d0a2 P 90d18fee8d86490cb8a7c520c736697b [term 1 FOLLOWER]: Refusing update from remote peer 3175b569f4f7467f92f0893d6729efb1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:06.377173 23239 consensus_queue.cc:1035] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0889709de474fc68bf6ac445c3e9d8d" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 42163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:06.377410 23234 consensus_queue.cc:1035] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d18fee8d86490cb8a7c520c736697b" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:06.385058 23242 mvcc.cc:205] Tried to move back new op lower bound from 7298808079868985344 to 7298808079713148928. Current Snapshot: MvccSnapshot[applied={T|T < 7298808079868985344}]
I20260620 06:06:06.385701 23243 mvcc.cc:205] Tried to move back new op lower bound from 7298808079868985344 to 7298808079713148928. Current Snapshot: MvccSnapshot[applied={T|T < 7298808079868985344}]
I20260620 06:06:06.394425 22752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:06.394656 22752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:06.397537 22752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6ead750e11a420c85e2a5a93742fc65: Sending DeleteTablet for 3 replicas of tablet 1e6e1eccddc74f3a899337acc504d0a2
I20260620 06:06:06.398485 22878 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e6e1eccddc74f3a899337acc504d0a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:06 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:41026
I20260620 06:06:06.398484 23146 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e6e1eccddc74f3a899337acc504d0a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:06 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:33250
I20260620 06:06:06.398533 23012 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e6e1eccddc74f3a899337acc504d0a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:06 UTC) from {username='kudu', principal='kudu/127.3.115.62@KRBTEST.COM'} at 127.0.0.1:51290
I20260620 06:06:06.399097 23252 tablet_replica.cc:330] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: stopping tablet replica
I20260620 06:06:06.399385 23251 tablet_replica.cc:330] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: stopping tablet replica
I20260620 06:06:06.399712 23252 raft_consensus.cc:2229] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 06:06:06.399770 23251 raft_consensus.cc:2229] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:06.400031 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22815
I20260620 06:06:06.400097 23252 raft_consensus.cc:2258] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:06.400127 23251 raft_consensus.cc:2258] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:06.402151 23252 ts_tablet_manager.cc:1905] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:06.405179 23252 ts_tablet_manager.cc:1918] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:06.405300 23252 log.cc:1196] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/1e6e1eccddc74f3a899337acc504d0a2
I20260620 06:06:06.405686 23252 ts_tablet_manager.cc:1939] T 1e6e1eccddc74f3a899337acc504d0a2 P 3175b569f4f7467f92f0893d6729efb1: Deleting consensus metadata
I20260620 06:06:06.406369 23251 ts_tablet_manager.cc:1905] T 1e6e1eccddc74f3a899337acc504d0a2 P f0889709de474fc68bf6ac445c3e9d8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:06.407017 22740 catalog_manager.cc:4841] TS 3175b569f4f7467f92f0893d6729efb1 (127.3.115.3:39321): tablet 1e6e1eccddc74f3a899337acc504d0a2 (table test-table [id=4c13f07e79d1439ba96e3062f79fa1c3]) successfully deleted
W20260620 06:06:06.408633 22739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:35187 (error 108)
I20260620 06:06:06.408846 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22949
I20260620 06:06:06.410259 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.409933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:35187 (local address 127.0.0.1:41040)
0620 06:06:06.410084 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.410156 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260620 06:06:06.410444 22739 catalog_manager.cc:4584] TS 90d18fee8d86490cb8a7c520c736697b (127.3.115.1:35187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1e6e1eccddc74f3a899337acc504d0a2: Network error: Client connection negotiation failed: client connection to 127.3.115.1:35187: connect: Connection refused (error 111)
I20260620 06:06:06.416510 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23083
I20260620 06:06:06.420892 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.416413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:42163 (local address 127.0.0.1:51292)
0620 06:06:06.416549 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.416618 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.2:42163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 06:06:06.422847 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 22721
2026-06-20T06:06:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2471 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T06:06:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:06Z Disabled control of system clock
I20260620 06:06:06.454545 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42107
--webserver_interface=127.3.115.62
--webserver_port=0
--builtin_ntp_servers=127.3.115.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:06:06.570930 23258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:06.571245 23258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:06.571287 23258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:06.575379 23258 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:06.575459 23258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:06.575474 23258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:06.575491 23258 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:06.575506 23258 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:06.580526 23258 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42107
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:06.581655 23258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:06.583045 23258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:06.591388 23264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:06.591389 23263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:06.591550 23258 server_base.cc:1034] running on GCE node
W20260620 06:06:06.591388 23266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:06.592312 23258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:06.593600 23258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:06.594786 23258 hybrid_clock.cc:648] HybridClock initialized: now 1781935566594757 us; error 47 us; skew 500 ppm
I20260620 06:06:06.596982 23258 webserver.cc:460] Webserver started at http://127.3.115.62:36523/ using document root <none> and password file <none>
I20260620 06:06:06.597621 23258 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:06.597728 23258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:06.597970 23258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:06.600220 23258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "e63793ecae484e37beb4fb0f02631542"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "0d0e3e3c3105c2b71f421edb6bff9a5c"
server_key_iv: "3f2340a53805e824c3c5f829b58f2ddc"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.600890 23258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "e63793ecae484e37beb4fb0f02631542"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "0d0e3e3c3105c2b71f421edb6bff9a5c"
server_key_iv: "3f2340a53805e824c3c5f829b58f2ddc"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.604910 23258 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260620 06:06:06.607409 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.608601 23258 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:06.608789 23258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "e63793ecae484e37beb4fb0f02631542"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "0d0e3e3c3105c2b71f421edb6bff9a5c"
server_key_iv: "3f2340a53805e824c3c5f829b58f2ddc"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.608906 23258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:06.622924 23258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:06.623687 23258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:06.623907 23258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:06.631234 23258 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42107
I20260620 06:06:06.631238 23324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42107 every 8 connection(s)
I20260620 06:06:06.632227 23258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:06.635449 23325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:06.640965 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23258
I20260620 06:06:06.641088 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:06.641396 3532 external_mini_cluster.cc:1428] Setting key 272414161b2fe89d356834f141d5b076
I20260620 06:06:06.641388 23325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542: Bootstrap starting.
I20260620 06:06:06.644039 23325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:06.644871 23325 log.cc:824] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:06.646941 23325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542: No bootstrap required, opened a new log
I20260620 06:06:06.650767 23325 raft_consensus.cc:348] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63793ecae484e37beb4fb0f02631542" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42107 } }
I20260620 06:06:06.650938 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.642842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.3.115.62:42107)
0620 06:06:06.643485 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.643499 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.643536 (+ 37us) server_negotiation.cc:407] Connection header received
0620 06:06:06.644265 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.644285 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.644761 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.645137 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:06.645679 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.646647 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.647596 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.648080 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.648267 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.649140 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:06.649150 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:06.649164 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:06.649565 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:06.649621 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:06.649642 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:06.649774 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.649921 (+ 147us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.650185 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":135,"threads_started":1}
I20260620 06:06:06.651031 23325 raft_consensus.cc:374] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:06.651119 23325 raft_consensus.cc:729] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e63793ecae484e37beb4fb0f02631542, State: Initialized, Role: FOLLOWER
I20260620 06:06:06.651652 23325 consensus_queue.cc:260] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63793ecae484e37beb4fb0f02631542" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42107 } }
I20260620 06:06:06.651815 23325 raft_consensus.cc:388] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:06.651878 23325 raft_consensus.cc:482] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:06.651969 23325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:06.652976 23325 raft_consensus.cc:504] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63793ecae484e37beb4fb0f02631542" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42107 } }
I20260620 06:06:06.653378 23325 leader_election.cc:302] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e63793ecae484e37beb4fb0f02631542; no voters:
I20260620 06:06:06.653756 23325 leader_election.cc:288] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:06.653870 23330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:06.654109 23330 raft_consensus.cc:686] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [term 1 LEADER]: Becoming Leader. State: Replica: e63793ecae484e37beb4fb0f02631542, State: Running, Role: LEADER
I20260620 06:06:06.654431 23330 consensus_queue.cc:237] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63793ecae484e37beb4fb0f02631542" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42107 } }
I20260620 06:06:06.654980 23325 sys_catalog.cc:564] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:06.656248 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e63793ecae484e37beb4fb0f02631542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63793ecae484e37beb4fb0f02631542" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42107 } } }
I20260620 06:06:06.656263 23332 sys_catalog.cc:455] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e63793ecae484e37beb4fb0f02631542. Latest consensus state: current_term: 1 leader_uuid: "e63793ecae484e37beb4fb0f02631542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e63793ecae484e37beb4fb0f02631542" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42107 } } }
I20260620 06:06:06.656394 23332 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:06.656394 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:06.656929 23339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:06.660152 23339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:06.666164 23339 catalog_manager.cc:1347] Generated new cluster ID: 280163c7bee84f93b474e92c5ae32491
I20260620 06:06:06.666276 23339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:06.679325 23339 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:06.680711 23339 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:06.694492 23339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542: Generated new TSK 0
I20260620 06:06:06.695343 23339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:06.698520 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.1:0
--local_ip_for_outbound_sockets=127.3.115.1
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42107
--builtin_ntp_servers=127.3.115.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 06:06:06.815002 23349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:06.815280 23349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:06.815320 23349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:06.819309 23349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:06.819371 23349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:06.819427 23349 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.1
I20260620 06:06:06.824158 23349 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.115.1
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.1
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:06.825306 23349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:06.826643 23349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:06.836603 23357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:06.836795 23349 server_base.cc:1034] running on GCE node
W20260620 06:06:06.836589 23354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:06.836589 23355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:06.837525 23349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:06.838276 23349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:06.839437 23349 hybrid_clock.cc:648] HybridClock initialized: now 1781935566839407 us; error 72 us; skew 500 ppm
I20260620 06:06:06.841500 23349 webserver.cc:460] Webserver started at http://127.3.115.1:37817/ using document root <none> and password file <none>
I20260620 06:06:06.842114 23349 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:06.842181 23349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:06.842417 23349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:06.844590 23349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data/instance:
uuid: "bd2017642b384c388a2926b9c8b05896"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "13fb21a6adfcd1a1c3a6b7d7916d0ad0"
server_key_iv: "8488685705415a08cd80c59b96c4ced6"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.845300 23349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance:
uuid: "bd2017642b384c388a2926b9c8b05896"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "13fb21a6adfcd1a1c3a6b7d7916d0ad0"
server_key_iv: "8488685705415a08cd80c59b96c4ced6"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.849262 23349 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:06.851922 23363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.853317 23349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:06.853458 23349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal
uuid: "bd2017642b384c388a2926b9c8b05896"
format_stamp: "Formatted at 2026-06-20 06:06:06 on dist-test-slave-4h14"
server_key: "13fb21a6adfcd1a1c3a6b7d7916d0ad0"
server_key_iv: "8488685705415a08cd80c59b96c4ced6"
server_key_version: "encryptionkey@0"
I20260620 06:06:06.853582 23349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:06.880971 23349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:06.881745 23349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:06.881953 23349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:06.882580 23349 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:06.883563 23349 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:06.883635 23349 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.883708 23349 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:06.883754 23349 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:06.892539 23349 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.1:33701
I20260620 06:06:06.892594 23476 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.1:33701 every 8 connection(s)
I20260620 06:06:06.893530 23349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/data/info.pb
I20260620 06:06:06.895788 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23349
I20260620 06:06:06.895903 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-0/wal/instance
I20260620 06:06:06.896193 3532 external_mini_cluster.cc:1428] Setting key 39d10b8c87d6fb8be98c9dfdbb4720fa
I20260620 06:06:06.898141 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.2:0
--local_ip_for_outbound_sockets=127.3.115.2
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42107
--builtin_ntp_servers=127.3.115.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:06:06.903596 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.895359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:40307 (local address 127.3.115.62:42107)
0620 06:06:06.895485 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:06.895489 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:06.896684 (+ 1195us) server_negotiation.cc:407] Connection header received
0620 06:06:06.898088 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:06.898092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:06.898147 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:06.898254 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:06.899240 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.899846 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.901290 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.901722 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.901858 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.902988 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:06.902992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:06.902997 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:06.903017 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:06.903045 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:06.903061 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:06.903154 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:06.903299 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 06:06:06.903377 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 06:06:06.904376 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:06.895722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42107 (local address 127.3.115.1:40307)
0620 06:06:06.896474 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:06.896527 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:06.897878 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:06.898467 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:06.898474 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:06.898562 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:06.899088 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.899102 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.899985 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.899990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.901170 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:06.901180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:06.901881 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:06.901884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:06.902028 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:06.902720 (+ 692us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:06:06.902721 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:06.902764 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:06:06.902830 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:06.903159 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:06.903166 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:06.903170 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 06:06:06.903270 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 06:06:06.903499 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":160,"threads_started":1}
I20260620 06:06:06.905792 23477 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42107
I20260620 06:06:06.906090 23477 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:06.906847 23477 heartbeater.cc:511] Master 127.3.115.62:42107 requested a full tablet report, sending...
I20260620 06:06:06.908758 23289 ts_manager.cc:194] Registered new tserver with Master: bd2017642b384c388a2926b9c8b05896 (127.3.115.1:33701)
I20260620 06:06:06.909929 23289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.1:40307
W20260620 06:06:07.021375 23480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:07.021696 23480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:07.021739 23480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:07.026171 23480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:07.026247 23480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:07.026307 23480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.2
I20260620 06:06:07.031533 23480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.115.2
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.2
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:07.032883 23480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:07.034687 23480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:07.045197 23485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:07.045230 23488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:07.045321 23486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:07.045419 23480 server_base.cc:1034] running on GCE node
I20260620 06:06:07.046018 23480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:07.046841 23480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:07.048027 23480 hybrid_clock.cc:648] HybridClock initialized: now 1781935567048009 us; error 33 us; skew 500 ppm
I20260620 06:06:07.050068 23480 webserver.cc:460] Webserver started at http://127.3.115.2:41479/ using document root <none> and password file <none>
I20260620 06:06:07.050676 23480 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:07.050770 23480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:07.051013 23480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:07.053262 23480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data/instance:
uuid: "dc7d784e24aa42a2bcd0cb3c69776af9"
format_stamp: "Formatted at 2026-06-20 06:06:07 on dist-test-slave-4h14"
server_key: "76635cfeb23d8ce7f9c0869105954cdf"
server_key_iv: "9e3756396f243da998a449d17536684f"
server_key_version: "encryptionkey@0"
I20260620 06:06:07.053864 23480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance:
uuid: "dc7d784e24aa42a2bcd0cb3c69776af9"
format_stamp: "Formatted at 2026-06-20 06:06:07 on dist-test-slave-4h14"
server_key: "76635cfeb23d8ce7f9c0869105954cdf"
server_key_iv: "9e3756396f243da998a449d17536684f"
server_key_version: "encryptionkey@0"
I20260620 06:06:07.058266 23480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:07.060882 23494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:07.062364 23480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:07.062508 23480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal
uuid: "dc7d784e24aa42a2bcd0cb3c69776af9"
format_stamp: "Formatted at 2026-06-20 06:06:07 on dist-test-slave-4h14"
server_key: "76635cfeb23d8ce7f9c0869105954cdf"
server_key_iv: "9e3756396f243da998a449d17536684f"
server_key_version: "encryptionkey@0"
I20260620 06:06:07.062630 23480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:07.074617 23480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:07.075407 23480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:07.075614 23480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:07.076222 23480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:07.077288 23480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:07.077361 23480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:07.077431 23480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:07.077483 23480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:07.087198 23480 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.2:39807
I20260620 06:06:07.087224 23607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.2:39807 every 8 connection(s)
I20260620 06:06:07.088250 23480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/data/info.pb
I20260620 06:06:07.095432 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23480
I20260620 06:06:07.095604 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal/instance
I20260620 06:06:07.095956 3532 external_mini_cluster.cc:1428] Setting key 5c4976d49817a6cdd3eaacbb2fbf66f5
I20260620 06:06:07.098243 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.115.3:0
--local_ip_for_outbound_sockets=127.3.115.3
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42107
--builtin_ntp_servers=127.3.115.20:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 06:06:07.098798 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.090348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.2:60993 (local address 127.3.115.62:42107)
0620 06:06:07.090497 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.090501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.091631 (+ 1130us) server_negotiation.cc:407] Connection header received
0620 06:06:07.093190 (+ 1559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.093194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.093266 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.093372 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:07.094325 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.094893 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.096258 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.096491 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.096631 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.098185 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:07.098190 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:07.098193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:07.098209 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:07.098235 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:07.098244 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:07.098329 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.098562 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.098633 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:06:07.099825 23610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.090733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42107 (local address 127.3.115.2:60993)
0620 06:06:07.091446 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.091488 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.092934 (+ 1446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.093566 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.093573 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.093664 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:07.094174 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.094186 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.095031 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.095044 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.096135 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.096145 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.096788 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.096799 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.097083 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.097897 (+ 814us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:06:07.097898 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:07.097942 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:06:07.098012 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:07.098332 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:07.098345 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:07.098354 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.098559 (+ 205us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.098879 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":122,"threads_started":1}
I20260620 06:06:07.101251 23608 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42107
I20260620 06:06:07.101527 23608 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:07.102252 23608 heartbeater.cc:511] Master 127.3.115.62:42107 requested a full tablet report, sending...
I20260620 06:06:07.103493 23289 ts_manager.cc:194] Registered new tserver with Master: dc7d784e24aa42a2bcd0cb3c69776af9 (127.3.115.2:39807)
I20260620 06:06:07.104099 23289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.2:60993
W20260620 06:06:07.215389 23611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:07.215659 23611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:07.215698 23611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:07.219910 23611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:07.219986 23611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:07.220043 23611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.115.3
I20260620 06:06:07.225018 23611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.115.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.115.3
--webserver_port=0
--tserver_master_addrs=127.3.115.62:42107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.115.3
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:07.226171 23611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:07.227591 23611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:07.237506 23619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:07.237468 23616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:07.237510 23617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:07.237828 23611 server_base.cc:1034] running on GCE node
I20260620 06:06:07.238461 23611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:07.239162 23611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:07.240337 23611 hybrid_clock.cc:648] HybridClock initialized: now 1781935567240307 us; error 53 us; skew 500 ppm
I20260620 06:06:07.242597 23611 webserver.cc:460] Webserver started at http://127.3.115.3:46087/ using document root <none> and password file <none>
I20260620 06:06:07.243217 23611 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:07.243283 23611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:07.243515 23611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:07.246536 23611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data/instance:
uuid: "de78545da3dc4184ab792de0de711ab9"
format_stamp: "Formatted at 2026-06-20 06:06:07 on dist-test-slave-4h14"
server_key: "67b5b64a97c59748bf6d97cee73f6d87"
server_key_iv: "16d8fb06af6571b9ba5871dd441ab5b2"
server_key_version: "encryptionkey@0"
I20260620 06:06:07.247255 23611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance:
uuid: "de78545da3dc4184ab792de0de711ab9"
format_stamp: "Formatted at 2026-06-20 06:06:07 on dist-test-slave-4h14"
server_key: "67b5b64a97c59748bf6d97cee73f6d87"
server_key_iv: "16d8fb06af6571b9ba5871dd441ab5b2"
server_key_version: "encryptionkey@0"
I20260620 06:06:07.251515 23611 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:07.254295 23625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:07.255859 23611 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:07.255992 23611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal
uuid: "de78545da3dc4184ab792de0de711ab9"
format_stamp: "Formatted at 2026-06-20 06:06:07 on dist-test-slave-4h14"
server_key: "67b5b64a97c59748bf6d97cee73f6d87"
server_key_iv: "16d8fb06af6571b9ba5871dd441ab5b2"
server_key_version: "encryptionkey@0"
I20260620 06:06:07.256109 23611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:07.273684 23611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:07.274427 23611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:07.274648 23611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:07.275267 23611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 06:06:07.276235 23611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 06:06:07.276283 23611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:07.276352 23611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 06:06:07.276386 23611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:07.287042 23611 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.3:33689
I20260620 06:06:07.287149 23738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.3:33689 every 8 connection(s)
I20260620 06:06:07.288244 23611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/data/info.pb
I20260620 06:06:07.296383 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23611
I20260620 06:06:07.296516 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal/instance
I20260620 06:06:07.296839 3532 external_mini_cluster.cc:1428] Setting key 4d9f9c60bdefbd629547bde4cd1547ad
I20260620 06:06:07.299302 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.290563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.3:49483 (local address 127.3.115.62:42107)
0620 06:06:07.290752 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.290757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.292156 (+ 1399us) server_negotiation.cc:407] Connection header received
0620 06:06:07.293756 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.293759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.293810 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.293894 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:07.294852 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.295417 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.297091 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.297335 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.297495 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.298766 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:07.298770 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:07.298773 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:07.298788 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:07.298814 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:07.298828 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:07.298920 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.299098 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.299167 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260620 06:06:07.300421 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.291064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:42107 (local address 127.3.115.3:49483)
0620 06:06:07.291989 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.292037 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.293513 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.294048 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.294056 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.294156 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:07.294712 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.294725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.295572 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.295576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.296867 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.296879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.297542 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.297545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.297696 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.298429 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 06:06:07.298431 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:07.298497 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 06:06:07.298593 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:07.298921 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:07.298934 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:07.298943 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.299092 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.299478 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":129,"threads_started":1}
I20260620 06:06:07.301613 23739 heartbeater.cc:347] Connected to a master server at 127.3.115.62:42107
I20260620 06:06:07.301906 23739 heartbeater.cc:464] Registering TS with master...
I20260620 06:06:07.302485 23739 heartbeater.cc:511] Master 127.3.115.62:42107 requested a full tablet report, sending...
I20260620 06:06:07.303617 23289 ts_manager.cc:194] Registered new tserver with Master: de78545da3dc4184ab792de0de711ab9 (127.3.115.3:33689)
I20260620 06:06:07.304276 23289 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.115.3:49483
I20260620 06:06:07.311275 3532 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 06:06:07.317584 23328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.313508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.3.115.62:42107)
0620 06:06:07.313663 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.313668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.313827 (+ 159us) server_negotiation.cc:407] Connection header received
0620 06:06:07.314007 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.314010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.314066 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.314154 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:07.314445 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.314986 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.315860 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.316099 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.316211 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.317073 (+ 862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:07.317077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:07.317079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:07.317093 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:07.317130 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:07.317138 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:07.317219 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.317340 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.317401 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:06:07.320950 23289 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 06:06:07.324124 23289 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 06:06:07.339689 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.333819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.3.115.2:39807)
0620 06:06:07.334829 (+ 1010us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.334845 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.334873 (+ 28us) server_negotiation.cc:407] Connection header received
0620 06:06:07.334957 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.334965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.335194 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.335720 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:07.335870 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.336782 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.338259 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.338987 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.339125 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.339349 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.339417 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.339516 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":153,"threads_started":1}
I20260620 06:06:07.339684 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.333634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39807 (local address 127.0.0.1:42066)
0620 06:06:07.334039 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.334097 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.334228 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.335371 (+ 1143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.335377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.335400 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:07.335583 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.335591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.336897 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.336900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.338115 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.338125 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.339122 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.339126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.339232 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.339269 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.339379 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.339509 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":135,"threads_started":1}
I20260620 06:06:07.340349 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.334496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:33701 (local address 127.0.0.1:56698)
0620 06:06:07.334916 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.334933 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.335036 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.335856 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.335859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.335876 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:07.335965 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.335971 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.337524 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.337533 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.338749 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.338761 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.339920 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.339928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.340017 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.340045 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.340094 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.340185 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":109,"threads_started":1}
I20260620 06:06:07.340772 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.333888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:33689 (local address 127.0.0.1:59028)
0620 06:06:07.334808 (+ 920us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.334823 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.334915 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.335599 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.335603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.335625 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:07.335743 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.335756 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.337440 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.337444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.338559 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.338569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.340270 (+ 1701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.340274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.340363 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.340384 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.340538 (+ 154us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.340618 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260620 06:06:07.341142 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.334670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.3.115.1:33701)
0620 06:06:07.335331 (+ 661us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.335347 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.335371 (+ 24us) server_negotiation.cc:407] Connection header received
0620 06:06:07.335457 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.335462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.335689 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.335867 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:07.336105 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.337347 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.338902 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.339745 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.340577 (+ 832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.340795 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.340846 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.340960 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":94,"threads_started":1}
I20260620 06:06:07.341583 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.334420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59028 (local address 127.3.115.3:33689)
0620 06:06:07.335187 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.335199 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.335220 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:06:07.335291 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.335296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.335468 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.335633 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:07.335896 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.337304 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.338697 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.340135 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.340271 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.340731 (+ 460us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.340800 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.341415 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":190,"threads_started":1}
I20260620 06:06:07.343075 23542 tablet_service.cc:1467] Processing CreateTablet for tablet 9284f06643614b438b3a790d5f1dacca (DEFAULT_TABLE table=test-table [id=4b897656807846a4ba5eda45c6d18b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:07.344080 23542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9284f06643614b438b3a790d5f1dacca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:07.344048 23411 tablet_service.cc:1467] Processing CreateTablet for tablet 9284f06643614b438b3a790d5f1dacca (DEFAULT_TABLE table=test-table [id=4b897656807846a4ba5eda45c6d18b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:07.344481 23673 tablet_service.cc:1467] Processing CreateTablet for tablet 9284f06643614b438b3a790d5f1dacca (DEFAULT_TABLE table=test-table [id=4b897656807846a4ba5eda45c6d18b4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 06:06:07.345278 23411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9284f06643614b438b3a790d5f1dacca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:07.345573 23673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9284f06643614b438b3a790d5f1dacca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:07.351099 23756 tablet_bootstrap.cc:492] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Bootstrap starting.
I20260620 06:06:07.353248 23757 tablet_bootstrap.cc:492] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: Bootstrap starting.
I20260620 06:06:07.353521 23756 tablet_bootstrap.cc:654] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:07.354490 23756 log.cc:824] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:07.355262 23758 tablet_bootstrap.cc:492] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Bootstrap starting.
I20260620 06:06:07.355468 23757 tablet_bootstrap.cc:654] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:07.356294 23757 log.cc:824] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:07.357108 23756 tablet_bootstrap.cc:492] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: No bootstrap required, opened a new log
I20260620 06:06:07.357402 23756 ts_tablet_manager.cc:1397] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:06:07.357651 23758 tablet_bootstrap.cc:654] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:07.358242 23757 tablet_bootstrap.cc:492] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: No bootstrap required, opened a new log
I20260620 06:06:07.358428 23757 ts_tablet_manager.cc:1397] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260620 06:06:07.358476 23758 log.cc:824] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:07.360533 23758 tablet_bootstrap.cc:492] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: No bootstrap required, opened a new log
I20260620 06:06:07.360762 23758 ts_tablet_manager.cc:1397] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260620 06:06:07.361836 23756 raft_consensus.cc:348] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.362094 23756 raft_consensus.cc:374] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:07.362164 23756 raft_consensus.cc:729] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc7d784e24aa42a2bcd0cb3c69776af9, State: Initialized, Role: FOLLOWER
I20260620 06:06:07.362650 23756 consensus_queue.cc:260] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.362697 23757 raft_consensus.cc:348] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.362936 23757 raft_consensus.cc:374] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:07.362986 23757 raft_consensus.cc:729] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd2017642b384c388a2926b9c8b05896, State: Initialized, Role: FOLLOWER
I20260620 06:06:07.363546 23757 consensus_queue.cc:260] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.363785 23756 ts_tablet_manager.cc:1428] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260620 06:06:07.364197 23608 heartbeater.cc:503] Master 127.3.115.62:42107 was elected leader, sending a full tablet report...
I20260620 06:06:07.364491 23757 ts_tablet_manager.cc:1428] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260620 06:06:07.364852 23477 heartbeater.cc:503] Master 127.3.115.62:42107 was elected leader, sending a full tablet report...
I20260620 06:06:07.365084 23758 raft_consensus.cc:348] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.365327 23758 raft_consensus.cc:374] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:07.365382 23758 raft_consensus.cc:729] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de78545da3dc4184ab792de0de711ab9, State: Initialized, Role: FOLLOWER
I20260620 06:06:07.365840 23758 consensus_queue.cc:260] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.366483 23739 heartbeater.cc:503] Master 127.3.115.62:42107 was elected leader, sending a full tablet report...
I20260620 06:06:07.366780 23758 ts_tablet_manager.cc:1428] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260620 06:06:07.394798 23478 tablet.cc:2369] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:07.517023 23763 raft_consensus.cc:482] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 06:06:07.517305 23763 raft_consensus.cc:504] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.518633 23763 leader_election.cc:288] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc7d784e24aa42a2bcd0cb3c69776af9 (127.3.115.2:39807), de78545da3dc4184ab792de0de711ab9 (127.3.115.3:33689)
I20260620 06:06:07.524124 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.519155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37527 (local address 127.3.115.3:33689)
0620 06:06:07.519314 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.519318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.519458 (+ 140us) server_negotiation.cc:407] Connection header received
0620 06:06:07.519566 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.519569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.519624 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.519716 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:07.520173 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.521038 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.522857 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.523510 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.523622 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.523698 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.523919 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.524014 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 06:06:07.524281 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.518972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.3:33689 (local address 127.3.115.1:37527)
0620 06:06:07.519366 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.519378 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.519479 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.519751 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.519755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.519807 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:07.520014 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.520025 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.521172 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.521176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.522745 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.522751 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.523659 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.523662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.523751 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.523772 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.523885 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.524071 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"mutex_wait_us":273,"spinlock_wait_cycles":25088,"thread_start_us":146,"threads_started":1}
I20260620 06:06:07.524530 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.518991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.115.1:37999 (local address 127.3.115.2:39807)
0620 06:06:07.519159 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.519167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.519357 (+ 190us) server_negotiation.cc:407] Connection header received
0620 06:06:07.519567 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.519570 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.519624 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.519716 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 06:06:07.520173 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.521038 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.523129 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.523760 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.523841 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.523930 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.524273 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.524368 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 06:06:07.524874 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.518869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.2:39807 (local address 127.3.115.1:37999)
0620 06:06:07.519273 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.519289 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:07.519412 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:07.519751 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:07.519755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:07.519806 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 06:06:07.520043 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.520050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.521172 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.521176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.522737 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:07.522751 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.523898 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.523901 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:07.524112 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.524135 (+ 23us) client_negotiation.cc:769] Sending connection context
0620 06:06:07.524693 (+ 558us) client_negotiation.cc:240] Negotiation successful
0620 06:06:07.524772 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":168,"spinlock_wait_cycles":30720,"thread_start_us":125,"threads_started":1}
I20260620 06:06:07.525028 23691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9284f06643614b438b3a790d5f1dacca" candidate_uuid: "bd2017642b384c388a2926b9c8b05896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de78545da3dc4184ab792de0de711ab9" is_pre_election: true
I20260620 06:06:07.525434 23691 raft_consensus.cc:2454] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd2017642b384c388a2926b9c8b05896 in term 0.
I20260620 06:06:07.525568 23562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9284f06643614b438b3a790d5f1dacca" candidate_uuid: "bd2017642b384c388a2926b9c8b05896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" is_pre_election: true
I20260620 06:06:07.525854 23562 raft_consensus.cc:2454] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd2017642b384c388a2926b9c8b05896 in term 0.
I20260620 06:06:07.525979 23367 leader_election.cc:302] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd2017642b384c388a2926b9c8b05896, de78545da3dc4184ab792de0de711ab9; no voters:
I20260620 06:06:07.526302 23763 raft_consensus.cc:2781] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 06:06:07.526398 23763 raft_consensus.cc:482] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 06:06:07.526453 23763 raft_consensus.cc:3037] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:07.527568 23763 raft_consensus.cc:504] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.527981 23763 leader_election.cc:288] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [CANDIDATE]: Term 1 election: Requested vote from peers dc7d784e24aa42a2bcd0cb3c69776af9 (127.3.115.2:39807), de78545da3dc4184ab792de0de711ab9 (127.3.115.3:33689)
I20260620 06:06:07.528385 23691 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9284f06643614b438b3a790d5f1dacca" candidate_uuid: "bd2017642b384c388a2926b9c8b05896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de78545da3dc4184ab792de0de711ab9"
I20260620 06:06:07.528411 23562 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9284f06643614b438b3a790d5f1dacca" candidate_uuid: "bd2017642b384c388a2926b9c8b05896" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9"
I20260620 06:06:07.528542 23691 raft_consensus.cc:3037] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:07.528563 23562 raft_consensus.cc:3037] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:07.529691 23562 raft_consensus.cc:2454] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2017642b384c388a2926b9c8b05896 in term 1.
I20260620 06:06:07.529685 23691 raft_consensus.cc:2454] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd2017642b384c388a2926b9c8b05896 in term 1.
I20260620 06:06:07.530036 23367 leader_election.cc:302] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd2017642b384c388a2926b9c8b05896, de78545da3dc4184ab792de0de711ab9; no voters:
I20260620 06:06:07.530230 23763 raft_consensus.cc:2781] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:07.530465 23763 raft_consensus.cc:686] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [term 1 LEADER]: Becoming Leader. State: Replica: bd2017642b384c388a2926b9c8b05896, State: Running, Role: LEADER
I20260620 06:06:07.530860 23763 consensus_queue.cc:237] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } }
I20260620 06:06:07.534474 23287 catalog_manager.cc:5495] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd2017642b384c388a2926b9c8b05896 (127.3.115.1). New cstate: current_term: 1 leader_uuid: "bd2017642b384c388a2926b9c8b05896" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd2017642b384c388a2926b9c8b05896" member_type: VOTER last_known_addr { host: "127.3.115.1" port: 33701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 } health_report { overall_health: UNKNOWN } } }
W20260620 06:06:07.539328 23740 tablet.cc:2369] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:07.577687 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.574315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.3.115.1:33701)
0620 06:06:07.574434 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:07.574439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:07.574462 (+ 23us) server_negotiation.cc:407] Connection header received
0620 06:06:07.574579 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:07.574581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:07.574643 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:07.574736 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:07.574949 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.575462 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.576258 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:07.576470 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:07.576565 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:07.576874 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:07.577347 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:07.577443 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:07.577502 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 06:06:07.577566 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 06:06:07.584066 23562 raft_consensus.cc:1261] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 1 FOLLOWER]: Refusing update from remote peer bd2017642b384c388a2926b9c8b05896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:07.584066 23691 raft_consensus.cc:1261] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 1 FOLLOWER]: Refusing update from remote peer bd2017642b384c388a2926b9c8b05896: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:07.584838 23763 consensus_queue.cc:1035] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc7d784e24aa42a2bcd0cb3c69776af9" member_type: VOTER last_known_addr { host: "127.3.115.2" port: 39807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:07.585089 23767 consensus_queue.cc:1035] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de78545da3dc4184ab792de0de711ab9" member_type: VOTER last_known_addr { host: "127.3.115.3" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 06:06:07.590049 23609 tablet.cc:2369] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 06:06:07.593910 23771 mvcc.cc:205] Tried to move back new op lower bound from 7298808084816592896 to 7298808084609208320. Current Snapshot: MvccSnapshot[applied={T|T < 7298808084816592896}]
I20260620 06:06:07.594553 23772 mvcc.cc:205] Tried to move back new op lower bound from 7298808084816592896 to 7298808084609208320. Current Snapshot: MvccSnapshot[applied={T|T < 7298808084816592896}]
I20260620 06:06:07.602856 23288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:07.603018 23288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 06:06:07.605497 23288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e63793ecae484e37beb4fb0f02631542: Sending DeleteTablet for 3 replicas of tablet 9284f06643614b438b3a790d5f1dacca
I20260620 06:06:07.606451 23542 tablet_service.cc:1514] Processing DeleteTablet for tablet 9284f06643614b438b3a790d5f1dacca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:07 UTC) from {username='slave'} at 127.0.0.1:42066
I20260620 06:06:07.606433 23673 tablet_service.cc:1514] Processing DeleteTablet for tablet 9284f06643614b438b3a790d5f1dacca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:07 UTC) from {username='slave'} at 127.0.0.1:59028
I20260620 06:06:07.606580 23411 tablet_service.cc:1514] Processing DeleteTablet for tablet 9284f06643614b438b3a790d5f1dacca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 06:06:07 UTC) from {username='slave'} at 127.0.0.1:56698
I20260620 06:06:07.606930 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23349
I20260620 06:06:07.607002 23783 tablet_replica.cc:330] T 9284f06643614b438b3a790d5f1dacca P bd2017642b384c388a2926b9c8b05896: stopping tablet replica
I20260620 06:06:07.607677 23782 tablet_replica.cc:330] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: stopping tablet replica
I20260620 06:06:07.607723 23781 tablet_replica.cc:330] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: stopping tablet replica
I20260620 06:06:07.608029 23781 raft_consensus.cc:2229] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:07.608029 23782 raft_consensus.cc:2229] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 06:06:07.608372 23781 raft_consensus.cc:2258] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:07.608372 23782 raft_consensus.cc:2258] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 06:06:07.610180 23782 ts_tablet_manager.cc:1905] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:07.610802 23781 ts_tablet_manager.cc:1905] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 06:06:07.613219 23781 ts_tablet_manager.cc:1918] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:07.613334 23781 log.cc:1196] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-1/wal/wals/9284f06643614b438b3a790d5f1dacca
I20260620 06:06:07.613633 23781 ts_tablet_manager.cc:1939] T 9284f06643614b438b3a790d5f1dacca P dc7d784e24aa42a2bcd0cb3c69776af9: Deleting consensus metadata
I20260620 06:06:07.614003 23782 ts_tablet_manager.cc:1918] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 06:06:07.614116 23782 log.cc:1196] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Deleting WAL directory at /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781935311537307-3532-0/minicluster-data/ts-2/wal/wals/9284f06643614b438b3a790d5f1dacca
I20260620 06:06:07.614444 23782 ts_tablet_manager.cc:1939] T 9284f06643614b438b3a790d5f1dacca P de78545da3dc4184ab792de0de711ab9: Deleting consensus metadata
I20260620 06:06:07.614930 23275 catalog_manager.cc:4841] TS dc7d784e24aa42a2bcd0cb3c69776af9 (127.3.115.2:39807): tablet 9284f06643614b438b3a790d5f1dacca (table test-table [id=4b897656807846a4ba5eda45c6d18b4c]) successfully deleted
I20260620 06:06:07.615630 23276 catalog_manager.cc:4841] TS de78545da3dc4184ab792de0de711ab9 (127.3.115.3:33689): tablet 9284f06643614b438b3a790d5f1dacca (table test-table [id=4b897656807846a4ba5eda45c6d18b4c]) successfully deleted
W20260620 06:06:07.615739 23275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.115.1:33701 (error 108)
I20260620 06:06:07.616532 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23480
I20260620 06:06:07.616840 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:07.616492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.1:33701 (local address 127.0.0.1:56710)
0620 06:06:07.616645 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:07.616744 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.1:33701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 06:06:07.617275 23275 catalog_manager.cc:4584] TS bd2017642b384c388a2926b9c8b05896 (127.3.115.1:33701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9284f06643614b438b3a790d5f1dacca: Network error: Client connection negotiation failed: client connection to 127.3.115.1:33701: connect: Connection refused (error 111)
I20260620 06:06:07.622666 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23611
I20260620 06:06:07.628616 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23258
2026-06-20T06:06:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1204 ms)
[----------] 6 tests from EncryptionPolicyTest (11026 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:07 dist-test-slave-4h14 krb5kdc[23787](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:07 dist-test-slave-4h14 krb5kdc[23787](info): set up 2 sockets
Jun 20 06:06:07 dist-test-slave-4h14 krb5kdc[23787](info): commencing operation
krb5kdc: starting...
W20260620 06:06:08.672999 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:08 dist-test-slave-4h14 krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:08.800559 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:08.916779 23800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:08.917240 23800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:08.917366 23800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:08.921559 23800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:08.921677 23800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:08.921720 23800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:08.921759 23800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:08.921790 23800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:08.927498 23800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45989
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:08.928754 23800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:08.929853 23800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:08.938944 23808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:08.938946 23805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:08.939059 23806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:08.939347 23800 server_base.cc:1034] running on GCE node
I20260620 06:06:08.939963 23800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:08.941306 23800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:08.942517 23800 hybrid_clock.cc:648] HybridClock initialized: now 1781935568942510 us; error 40 us; skew 500 ppm
Jun 20 06:06:08 dist-test-slave-4h14 krb5kdc[23787](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:08.945591 23800 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:08.946589 23800 webserver.cc:460] Webserver started at http://127.0.0.1:37537/ using document root <none> and password file <none>
I20260620 06:06:08.947220 23800 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:08.947285 23800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:08.947459 23800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:08.949640 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "dc906cd69efd4eb5a36541d2effbd85e"
format_stamp: "Formatted at 2026-06-20 06:06:08 on dist-test-slave-4h14"
server_key: "b53a4de1b0ccd86090741645ca428d37"
server_key_iv: "2c516fe515dee85300a7e5d032081e02"
server_key_version: "encryptionkey@0"
I20260620 06:06:08.950244 23800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "dc906cd69efd4eb5a36541d2effbd85e"
format_stamp: "Formatted at 2026-06-20 06:06:08 on dist-test-slave-4h14"
server_key: "b53a4de1b0ccd86090741645ca428d37"
server_key_iv: "2c516fe515dee85300a7e5d032081e02"
server_key_version: "encryptionkey@0"
I20260620 06:06:08.954387 23800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:08.957499 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:08.958968 23800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:08.959089 23800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "dc906cd69efd4eb5a36541d2effbd85e"
format_stamp: "Formatted at 2026-06-20 06:06:08 on dist-test-slave-4h14"
server_key: "b53a4de1b0ccd86090741645ca428d37"
server_key_iv: "2c516fe515dee85300a7e5d032081e02"
server_key_version: "encryptionkey@0"
I20260620 06:06:08.959192 23800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:08.989722 23800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:08.992861 23800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:08.993033 23800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:09.000615 23800 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45989
I20260620 06:06:09.000630 23867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45989 every 8 connection(s)
I20260620 06:06:09.001649 23800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:09.004529 23868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:09.007879 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23800
I20260620 06:06:09.008104 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:09.008430 3532 external_mini_cluster.cc:1428] Setting key 9f1067cb9ae6f24aba5e3c6fe068a71d
I20260620 06:06:09.010829 23868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e: Bootstrap starting.
I20260620 06:06:09.013167 23868 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:09.013913 23868 log.cc:824] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:09.015751 23868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e: No bootstrap required, opened a new log
Jun 20 06:06:09 dist-test-slave-4h14 krb5kdc[23787](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:06:09.019624 23868 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc906cd69efd4eb5a36541d2effbd85e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260620 06:06:09.019886 23868 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:09.019969 23868 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc906cd69efd4eb5a36541d2effbd85e, State: Initialized, Role: FOLLOWER
I20260620 06:06:09.020464 23868 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc906cd69efd4eb5a36541d2effbd85e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260620 06:06:09.020635 23868 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:09.020733 23868 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:09.020864 23868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:09.021816 23868 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc906cd69efd4eb5a36541d2effbd85e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260620 06:06:09.022182 23868 leader_election.cc:302] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc906cd69efd4eb5a36541d2effbd85e; no voters:
I20260620 06:06:09.022535 23868 leader_election.cc:288] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:09.022665 23873 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:09.022883 23873 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [term 1 LEADER]: Becoming Leader. State: Replica: dc906cd69efd4eb5a36541d2effbd85e, State: Running, Role: LEADER
I20260620 06:06:09.023260 23873 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc906cd69efd4eb5a36541d2effbd85e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } }
I20260620 06:06:09.023763 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:09.009905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58711 (local address 127.0.0.1:45989)
0620 06:06:09.010420 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:09.010451 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:09.010684 (+ 233us) server_negotiation.cc:407] Connection header received
0620 06:06:09.011517 (+ 833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:09.011536 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:09.011889 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:09.012250 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:09.013081 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:09.014212 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:09.015089 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:09.015582 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:09.015758 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:09.018382 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:09.018387 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:09.018402 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:09.018429 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:09.020369 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:09.020964 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:09.020970 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:09.020974 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:09.021092 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:09.021470 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:09.021473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:09.021474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:09.021877 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:09.022155 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:09.022526 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 06:06:09.022795 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":161,"threads_started":1}
I20260620 06:06:09.023924 23868 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:09.025290 23874 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc906cd69efd4eb5a36541d2effbd85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc906cd69efd4eb5a36541d2effbd85e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } } }
I20260620 06:06:09.025405 23874 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:09.025863 23875 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc906cd69efd4eb5a36541d2effbd85e. Latest consensus state: current_term: 1 leader_uuid: "dc906cd69efd4eb5a36541d2effbd85e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc906cd69efd4eb5a36541d2effbd85e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45989 } } }
I20260620 06:06:09.026005 23875 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:09.026378 23882 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:09.027503 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:09.029392 23882 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:09.035540 23882 catalog_manager.cc:1347] Generated new cluster ID: 0d9e8a0be650465cbf3912e138757eb8
I20260620 06:06:09.035629 23882 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:09.039352 23871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:09.030088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33887 (local address 127.0.0.1:45989)
0620 06:06:09.030199 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:09.030203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:09.030219 (+ 16us) server_negotiation.cc:407] Connection header received
0620 06:06:09.030349 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:09.030352 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:09.030399 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:09.030446 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:09.031412 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:09.032280 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:09.033446 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:09.033685 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:09.033806 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:09.035476 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:09.035480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:09.035483 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:09.035509 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:09.037441 (+ 1932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:09.037992 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:09.037996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:09.037999 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:09.038093 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:09.038437 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:09.038441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:09.038443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:09.038668 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:09.038819 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:09.039079 (+ 260us) server_negotiation.cc:299] Negotiation successful
0620 06:06:09.039184 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:06:09.043462 23882 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:09.044581 23882 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:09.058750 23882 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc906cd69efd4eb5a36541d2effbd85e: Generated new TSK 0
I20260620 06:06:09.059495 23882 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:09.064555 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23800
W20260620 06:06:09.071254 23785 connection.cc:535] client connection to 127.0.0.1:45989 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45989): Connection reset by peer (error 104)
2026-06-20T06:06:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1435 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:09 dist-test-slave-4h14 krb5kdc[23899](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:09 dist-test-slave-4h14 krb5kdc[23899](info): set up 2 sockets
Jun 20 06:06:09 dist-test-slave-4h14 krb5kdc[23899](info): commencing operation
krb5kdc: starting...
W20260620 06:06:10.128890 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:10 dist-test-slave-4h14 krb5kdc[23899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:10.252310 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33441
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36097
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33441
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:10.365772 23912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:10.366125 23912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:10.366209 23912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:10.370368 23912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:10.370457 23912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:10.370496 23912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:10.370555 23912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:10.370595 23912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:10.376077 23912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36097
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33441
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23912
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:10.377380 23912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:10.378499 23912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:10.387909 23920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:10.387925 23918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:10.388093 23912 server_base.cc:1034] running on GCE node
W20260620 06:06:10.388007 23917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:10.388790 23912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:10.389940 23912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:10.391108 23912 hybrid_clock.cc:648] HybridClock initialized: now 1781935570391079 us; error 44 us; skew 500 ppm
Jun 20 06:06:10 dist-test-slave-4h14 krb5kdc[23899](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:10.394045 23912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:10.394932 23912 webserver.cc:460] Webserver started at http://127.0.0.1:39731/ using document root <none> and password file <none>
I20260620 06:06:10.395468 23912 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:10.395526 23912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:10.395702 23912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:10.397810 23912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "cf74542397ee4bf1959582f4297dd556"
format_stamp: "Formatted at 2026-06-20 06:06:10 on dist-test-slave-4h14"
server_key: "0cc8082503e21bf8a1a2ef5ba7414328"
server_key_iv: "d924bf1298a1781a9858a94712f4aca9"
server_key_version: "encryptionkey@0"
I20260620 06:06:10.398371 23912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "cf74542397ee4bf1959582f4297dd556"
format_stamp: "Formatted at 2026-06-20 06:06:10 on dist-test-slave-4h14"
server_key: "0cc8082503e21bf8a1a2ef5ba7414328"
server_key_iv: "d924bf1298a1781a9858a94712f4aca9"
server_key_version: "encryptionkey@0"
I20260620 06:06:10.402637 23912 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:10.405253 23927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:10.406708 23912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:10.406870 23912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "cf74542397ee4bf1959582f4297dd556"
format_stamp: "Formatted at 2026-06-20 06:06:10 on dist-test-slave-4h14"
server_key: "0cc8082503e21bf8a1a2ef5ba7414328"
server_key_iv: "d924bf1298a1781a9858a94712f4aca9"
server_key_version: "encryptionkey@0"
I20260620 06:06:10.406988 23912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:10.430644 23912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:10.433990 23912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:10.434173 23912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:10.441779 23912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33441
I20260620 06:06:10.441793 23979 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33441 every 8 connection(s)
I20260620 06:06:10.442828 23912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:10.445739 23980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:10.449210 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 23912
I20260620 06:06:10.449357 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:10.449692 3532 external_mini_cluster.cc:1428] Setting key 26e2220f29c831d28b88c5718d6b6902
I20260620 06:06:10.452023 23980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556: Bootstrap starting.
I20260620 06:06:10.454267 23980 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:10.455102 23980 log.cc:824] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:10.457257 23980 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556: No bootstrap required, opened a new log
I20260620 06:06:10.459520 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:10.451162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38655 (local address 127.0.0.1:33441)
0620 06:06:10.451672 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:10.451688 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:10.451723 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:06:10.452410 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:10.452430 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:10.452866 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:10.453237 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:10.453671 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:10.454940 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:10.456084 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:10.456548 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:10.456746 (+ 198us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:10.457574 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:10.457579 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:10.457612 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:10.458003 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:10.458057 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:10.458079 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:10.458182 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:10.458411 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 06:06:10.458561 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":141,"threads_started":1}
I20260620 06:06:10.460397 23980 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf74542397ee4bf1959582f4297dd556" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260620 06:06:10.460690 23980 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:10.460759 23980 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf74542397ee4bf1959582f4297dd556, State: Initialized, Role: FOLLOWER
I20260620 06:06:10.461220 23980 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf74542397ee4bf1959582f4297dd556" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260620 06:06:10.461370 23980 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:10.461445 23980 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:10.461567 23980 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:10.462574 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:10.462785 23980 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf74542397ee4bf1959582f4297dd556" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260620 06:06:10.463120 23980 leader_election.cc:302] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf74542397ee4bf1959582f4297dd556; no voters:
I20260620 06:06:10.463451 23980 leader_election.cc:288] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:10.463522 23985 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:10.463744 23985 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [term 1 LEADER]: Becoming Leader. State: Replica: cf74542397ee4bf1959582f4297dd556, State: Running, Role: LEADER
I20260620 06:06:10.464044 23985 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf74542397ee4bf1959582f4297dd556" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } }
I20260620 06:06:10.464775 23980 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:10.465272 23985 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf74542397ee4bf1959582f4297dd556. Latest consensus state: current_term: 1 leader_uuid: "cf74542397ee4bf1959582f4297dd556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf74542397ee4bf1959582f4297dd556" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } } }
I20260620 06:06:10.465394 23985 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:10.465798 23986 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf74542397ee4bf1959582f4297dd556" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf74542397ee4bf1959582f4297dd556" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33441 } } }
I20260620 06:06:10.465962 23986 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:10.467228 23995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:10.470101 23983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:10.464805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45469 (local address 127.0.0.1:33441)
0620 06:06:10.464943 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:10.464948 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:10.465814 (+ 866us) server_negotiation.cc:407] Connection header received
0620 06:06:10.466017 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:10.466020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:10.466066 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:10.466115 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:10.466592 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:10.467386 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:10.468265 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:10.468546 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:10.468674 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:10.469607 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:10.469611 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:10.469614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:10.469628 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:10.469654 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:10.469668 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:10.469748 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:10.469938 (+ 190us) server_negotiation.cc:299] Negotiation successful
0620 06:06:10.469974 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:06:10.470739 23995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:10.476580 23995 catalog_manager.cc:1347] Generated new cluster ID: 9ba6770f94ab4234a69a5bcc68bd5954
I20260620 06:06:10.476678 23995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:10.497438 23995 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:10.498759 23995 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:10.506404 23995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf74542397ee4bf1959582f4297dd556: Generated new TSK 0
I20260620 06:06:10.507395 23995 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:10.521441 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 23912
W20260620 06:06:10.526738 23897 connection.cc:535] client connection to 127.0.0.1:33441 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33441): Connection reset by peer (error 104)
2026-06-20T06:06:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:10 dist-test-slave-4h14 krb5kdc[24011](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:10 dist-test-slave-4h14 krb5kdc[24011](info): set up 2 sockets
Jun 20 06:06:10 dist-test-slave-4h14 krb5kdc[24011](info): commencing operation
krb5kdc: starting...
W20260620 06:06:11.565075 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:11 dist-test-slave-4h14 krb5kdc[24011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:11.692735 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:11.809798 24024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:11.810127 24024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:11.810212 24024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:11.814399 24024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:11.814494 24024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:11.814533 24024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:11.814596 24024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:11.814635 24024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:11.820078 24024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:11.821296 24024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:11.822335 24024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:11.831085 24030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:11.831085 24029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:11.831171 24024 server_base.cc:1034] running on GCE node
W20260620 06:06:11.831115 24032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:11.831993 24024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:11.833184 24024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:11.834409 24024 hybrid_clock.cc:648] HybridClock initialized: now 1781935571834391 us; error 30 us; skew 500 ppm
Jun 20 06:06:11 dist-test-slave-4h14 krb5kdc[24011](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935571, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:11.837440 24024 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:11.838445 24024 webserver.cc:460] Webserver started at http://127.0.0.1:46443/ using document root <none> and password file <none>
I20260620 06:06:11.839006 24024 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:11.839067 24024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:11.839236 24024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:11.841331 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "2bab265a61fe40938ef63063cddfa971"
format_stamp: "Formatted at 2026-06-20 06:06:11 on dist-test-slave-4h14"
server_key: "d5eadbbe8e735ce85d02ce566a7d1605"
server_key_iv: "e63e8d4a3a2de42dd06ade170fb9e981"
server_key_version: "encryptionkey@0"
I20260620 06:06:11.841863 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "2bab265a61fe40938ef63063cddfa971"
format_stamp: "Formatted at 2026-06-20 06:06:11 on dist-test-slave-4h14"
server_key: "d5eadbbe8e735ce85d02ce566a7d1605"
server_key_iv: "e63e8d4a3a2de42dd06ade170fb9e981"
server_key_version: "encryptionkey@0"
I20260620 06:06:11.845589 24024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260620 06:06:11.848088 24039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:11.849396 24024 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:11.849541 24024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "2bab265a61fe40938ef63063cddfa971"
format_stamp: "Formatted at 2026-06-20 06:06:11 on dist-test-slave-4h14"
server_key: "d5eadbbe8e735ce85d02ce566a7d1605"
server_key_iv: "e63e8d4a3a2de42dd06ade170fb9e981"
server_key_version: "encryptionkey@0"
I20260620 06:06:11.849660 24024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:11.854475 24024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:11.857869 24024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:11.858088 24024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:11.865586 24024 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37641
I20260620 06:06:11.865592 24091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37641 every 8 connection(s)
I20260620 06:06:11.866533 24024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:11.869140 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24024
I20260620 06:06:11.869252 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:11.869406 24092 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:11.869602 3532 external_mini_cluster.cc:1428] Setting key ffc0f194a45976c27728e47c40573c2f
I20260620 06:06:11.875152 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:11.871028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57033 (local address 127.0.0.1:37641)
0620 06:06:11.871445 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:11.871464 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:11.871505 (+ 41us) server_negotiation.cc:407] Connection header received
0620 06:06:11.872247 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:11.872270 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:11.872637 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:11.873052 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:11.873767 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:11.873773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:11.873788 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:11.873799 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:11.873868 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:11.873895 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:11.874032 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:11.874105 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 06:06:11.874253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":107,"threads_started":1}
I20260620 06:06:11.876845 24092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971: Bootstrap starting.
I20260620 06:06:11.878185 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:11.879341 24092 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:11.880227 24092 log.cc:824] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:11.882372 24092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971: No bootstrap required, opened a new log
I20260620 06:06:11.886080 24095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:11.880429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48643 (local address 127.0.0.1:37641)
0620 06:06:11.880583 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:11.880588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:11.880798 (+ 210us) server_negotiation.cc:407] Connection header received
0620 06:06:11.880887 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:11.880890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:11.880925 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:11.881018 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:11.885633 (+ 4615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:11.885636 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:11.885638 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:11.885650 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:11.885678 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:11.885689 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:11.885766 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:11.885892 (+ 126us) server_negotiation.cc:299] Negotiation successful
0620 06:06:11.885929 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:11.886082 24092 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bab265a61fe40938ef63063cddfa971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260620 06:06:11.886504 24092 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:11.886556 24092 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2bab265a61fe40938ef63063cddfa971, State: Initialized, Role: FOLLOWER
I20260620 06:06:11.887032 24092 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bab265a61fe40938ef63063cddfa971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260620 06:06:11.887192 24092 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:11.887261 24092 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:11.887355 24092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:11.888212 24092 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bab265a61fe40938ef63063cddfa971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260620 06:06:11.888518 24092 leader_election.cc:302] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2bab265a61fe40938ef63063cddfa971; no voters:
I20260620 06:06:11.888850 24092 leader_election.cc:288] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:11.888942 24102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:11.889225 24102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [term 1 LEADER]: Becoming Leader. State: Replica: 2bab265a61fe40938ef63063cddfa971, State: Running, Role: LEADER
I20260620 06:06:11.889583 24102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bab265a61fe40938ef63063cddfa971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } }
I20260620 06:06:11.890148 24092 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:11.891276 24104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2bab265a61fe40938ef63063cddfa971. Latest consensus state: current_term: 1 leader_uuid: "2bab265a61fe40938ef63063cddfa971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bab265a61fe40938ef63063cddfa971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } } }
I20260620 06:06:11.891427 24104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:11.891728 24103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2bab265a61fe40938ef63063cddfa971" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2bab265a61fe40938ef63063cddfa971" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37641 } } }
I20260620 06:06:11.891825 24103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971 [sys.catalog]: This master's current role is: LEADER
W20260620 06:06:11.894842 24117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:11.894938 24117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:11.895028 24111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:11.895740 24111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:11.902026 24111 catalog_manager.cc:1347] Generated new cluster ID: 73d79c2f370b4f858ba3396884c865f1
I20260620 06:06:11.902333 24111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:11.921520 24111 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:11.921909 24111 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:11.939616 24111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2bab265a61fe40938ef63063cddfa971: Generated new TSK 0
I20260620 06:06:11.940851 24111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:11.954530 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24024
2026-06-20T06:06:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1433 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:12 dist-test-slave-4h14 krb5kdc[24123](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:12 dist-test-slave-4h14 krb5kdc[24123](info): set up 2 sockets
Jun 20 06:06:12 dist-test-slave-4h14 krb5kdc[24123](info): commencing operation
krb5kdc: starting...
W20260620 06:06:13.004987 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:13 dist-test-slave-4h14 krb5kdc[24123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:13.132537 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:13.243969 24136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:13.244331 24136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:13.244426 24136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:13.248596 24136 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:13.248750 24136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:13.248819 24136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:13.248859 24136 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:13.248896 24136 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:13.254381 24136 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:13.255569 24136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:13.256739 24136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:13.266202 24144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:13.266271 24141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:13.266202 24142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:13.266816 24136 server_base.cc:1034] running on GCE node
I20260620 06:06:13.267617 24136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:13.269369 24136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:13.270562 24136 hybrid_clock.cc:648] HybridClock initialized: now 1781935573270537 us; error 48 us; skew 500 ppm
Jun 20 06:06:13 dist-test-slave-4h14 krb5kdc[24123](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:13.273592 24136 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:13.274551 24136 webserver.cc:460] Webserver started at http://127.0.0.1:39067/ using document root <none> and password file <none>
I20260620 06:06:13.275089 24136 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:13.275149 24136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:13.275316 24136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:13.277534 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "6ed8640cc4b54733a3d428f29c0c4330"
format_stamp: "Formatted at 2026-06-20 06:06:13 on dist-test-slave-4h14"
server_key: "5fd19c354b7ce62be0727e0e2dd5f7aa"
server_key_iv: "3d530add38cc3808705acccb6be579fc"
server_key_version: "encryptionkey@0"
I20260620 06:06:13.278052 24136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "6ed8640cc4b54733a3d428f29c0c4330"
format_stamp: "Formatted at 2026-06-20 06:06:13 on dist-test-slave-4h14"
server_key: "5fd19c354b7ce62be0727e0e2dd5f7aa"
server_key_iv: "3d530add38cc3808705acccb6be579fc"
server_key_version: "encryptionkey@0"
I20260620 06:06:13.282804 24136 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260620 06:06:13.286455 24151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:13.288298 24136 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:13.288434 24136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "6ed8640cc4b54733a3d428f29c0c4330"
format_stamp: "Formatted at 2026-06-20 06:06:13 on dist-test-slave-4h14"
server_key: "5fd19c354b7ce62be0727e0e2dd5f7aa"
server_key_iv: "3d530add38cc3808705acccb6be579fc"
server_key_version: "encryptionkey@0"
I20260620 06:06:13.288586 24136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:13.300989 24136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:13.304028 24136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:13.304239 24136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:13.311510 24136 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42003
I20260620 06:06:13.311537 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42003 every 8 connection(s)
I20260620 06:06:13.312493 24136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:13.315325 24204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:13.319109 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24136
I20260620 06:06:13.319267 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:13.319626 3532 external_mini_cluster.cc:1428] Setting key 75fbb61f6156cc01ca58542407ffdd80
I20260620 06:06:13.321583 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330: Bootstrap starting.
I20260620 06:06:13.324206 24204 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:13.325074 24204 log.cc:824] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:13.327116 24204 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330: No bootstrap required, opened a new log
Jun 20 06:06:13 dist-test-slave-4h14 krb5kdc[24123](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:06:13.331223 24204 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed8640cc4b54733a3d428f29c0c4330" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42003 } }
I20260620 06:06:13.331496 24204 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:13.331585 24204 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ed8640cc4b54733a3d428f29c0c4330, State: Initialized, Role: FOLLOWER
I20260620 06:06:13.332113 24204 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed8640cc4b54733a3d428f29c0c4330" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42003 } }
I20260620 06:06:13.332275 24204 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:13.332347 24204 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:13.332471 24204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:13.333551 24204 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed8640cc4b54733a3d428f29c0c4330" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42003 } }
I20260620 06:06:13.333933 24204 leader_election.cc:302] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ed8640cc4b54733a3d428f29c0c4330; no voters:
I20260620 06:06:13.334295 24204 leader_election.cc:288] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:13.334424 24209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:13.334636 24209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [term 1 LEADER]: Becoming Leader. State: Replica: 6ed8640cc4b54733a3d428f29c0c4330, State: Running, Role: LEADER
I20260620 06:06:13.334794 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:13.321104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54969 (local address 127.0.0.1:42003)
0620 06:06:13.321563 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:13.321580 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:13.321618 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:06:13.322324 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:13.322342 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:13.322711 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:13.323003 (+ 292us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:13.323970 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:13.325117 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:13.326054 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:13.326547 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:13.326721 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:13.329288 (+ 2567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:13.329293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:13.329308 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:13.329336 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:13.331254 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:13.331821 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:13.331826 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:13.331831 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:13.331949 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:13.332270 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:13.332276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:13.332280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:13.332880 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:13.333108 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:13.333360 (+ 252us) server_negotiation.cc:299] Negotiation successful
0620 06:06:13.333715 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":141,"threads_started":1}
I20260620 06:06:13.334985 24209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed8640cc4b54733a3d428f29c0c4330" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42003 } }
I20260620 06:06:13.335650 24204 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:13.336066 24209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ed8640cc4b54733a3d428f29c0c4330. Latest consensus state: current_term: 1 leader_uuid: "6ed8640cc4b54733a3d428f29c0c4330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed8640cc4b54733a3d428f29c0c4330" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42003 } } }
I20260620 06:06:13.336164 24209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:13.336880 24213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:13.338600 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:13.338631 24210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ed8640cc4b54733a3d428f29c0c4330" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ed8640cc4b54733a3d428f29c0c4330" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42003 } } }
I20260620 06:06:13.338758 24210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:13.339946 24213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:13.346459 24213 catalog_manager.cc:1347] Generated new cluster ID: e5d11f55fd9c45328758bff29826e7d9
I20260620 06:06:13.346546 24213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:13.353446 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:13.341280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48143 (local address 127.0.0.1:42003)
0620 06:06:13.341419 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:13.341424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:13.342354 (+ 930us) server_negotiation.cc:407] Connection header received
0620 06:06:13.342483 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:13.342486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:13.342536 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:13.342617 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:13.343479 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:13.344245 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:13.345316 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:13.345574 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:13.345703 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:13.349797 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:13.349802 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:13.349805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:13.349835 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:13.351594 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:13.352100 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:13.352103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:13.352105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:13.352199 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:13.352477 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:13.352480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:13.352482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:13.352758 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:13.352884 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:13.353132 (+ 248us) server_negotiation.cc:299] Negotiation successful
0620 06:06:13.353275 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:06:13.361356 24213 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:13.362315 24213 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:13.367169 24213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ed8640cc4b54733a3d428f29c0c4330: Generated new TSK 0
I20260620 06:06:13.367980 24213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:13.372265 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24136
2026-06-20T06:06:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1416 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:13 dist-test-slave-4h14 krb5kdc[24235](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:13 dist-test-slave-4h14 krb5kdc[24235](info): set up 2 sockets
Jun 20 06:06:13 dist-test-slave-4h14 krb5kdc[24235](info): commencing operation
krb5kdc: starting...
W20260620 06:06:14.417038 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:14 dist-test-slave-4h14 krb5kdc[24235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:14.544039 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:14.659662 24248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:14.659948 24248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:14.660039 24248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:14.664255 24248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:14.664333 24248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:14.664359 24248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:14.664377 24248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:14.664422 24248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:14.670007 24248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:14.671160 24248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:14.672425 24248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:14.681509 24253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:14.681511 24254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:14.681722 24248 server_base.cc:1034] running on GCE node
W20260620 06:06:14.681509 24256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:14.682508 24248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:14.683779 24248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:14.684985 24248 hybrid_clock.cc:648] HybridClock initialized: now 1781935574684959 us; error 36 us; skew 500 ppm
Jun 20 06:06:14 dist-test-slave-4h14 krb5kdc[24235](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935574, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:14.687906 24248 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:14.688971 24248 webserver.cc:460] Webserver started at http://127.0.0.1:45921/ using document root <none> and password file <none>
I20260620 06:06:14.689560 24248 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:14.689656 24248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:14.689918 24248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:14.692122 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "2eb4735e89614d12aa568c20f6f4ccc6"
format_stamp: "Formatted at 2026-06-20 06:06:14 on dist-test-slave-4h14"
server_key: "810b00acfd811097e5c61db3ffda1260"
server_key_iv: "6282b2cff20f53812e72b4614e5f87c0"
server_key_version: "encryptionkey@0"
I20260620 06:06:14.692808 24248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "2eb4735e89614d12aa568c20f6f4ccc6"
format_stamp: "Formatted at 2026-06-20 06:06:14 on dist-test-slave-4h14"
server_key: "810b00acfd811097e5c61db3ffda1260"
server_key_iv: "6282b2cff20f53812e72b4614e5f87c0"
server_key_version: "encryptionkey@0"
I20260620 06:06:14.696805 24248 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:14.699425 24263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:14.700875 24248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:14.701033 24248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "2eb4735e89614d12aa568c20f6f4ccc6"
format_stamp: "Formatted at 2026-06-20 06:06:14 on dist-test-slave-4h14"
server_key: "810b00acfd811097e5c61db3ffda1260"
server_key_iv: "6282b2cff20f53812e72b4614e5f87c0"
server_key_version: "encryptionkey@0"
I20260620 06:06:14.701162 24248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:14.718248 24248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:14.721393 24248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:14.721611 24248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:14.729048 24248 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35853
I20260620 06:06:14.729067 24315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35853 every 8 connection(s)
I20260620 06:06:14.730046 24248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:14.730871 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24248
I20260620 06:06:14.731015 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:14.731308 3532 external_mini_cluster.cc:1428] Setting key ab212a86d7ab3abdcfec3799d5f0384a
I20260620 06:06:14.733171 24316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:14.738750 24316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6: Bootstrap starting.
I20260620 06:06:14.740319 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:14.732740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42791 (local address 127.0.0.1:35853)
0620 06:06:14.733175 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:14.733191 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:14.733226 (+ 35us) server_negotiation.cc:407] Connection header received
0620 06:06:14.733915 (+ 689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:14.733932 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:14.734284 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:14.734634 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:14.735050 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:14.736035 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:14.737005 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:14.737495 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:14.737672 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:14.738562 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:14.738567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:14.738580 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:14.739029 (+ 449us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:14.739084 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:14.739107 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:14.739216 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:14.739360 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 06:06:14.739576 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":151,"threads_started":1}
I20260620 06:06:14.741183 24316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:14.741919 24316 log.cc:824] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:14.743261 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:14.744172 24316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6: No bootstrap required, opened a new log
I20260620 06:06:14.749014 24316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 06:06:14.749359 24316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:14.749432 24316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb4735e89614d12aa568c20f6f4ccc6, State: Initialized, Role: FOLLOWER
I20260620 06:06:14.749717 24319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:14.745393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50995 (local address 127.0.0.1:35853)
0620 06:06:14.745499 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:14.745504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:14.745687 (+ 183us) server_negotiation.cc:407] Connection header received
0620 06:06:14.745786 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:14.745789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:14.745835 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:14.745923 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:14.746187 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:14.746996 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:14.747790 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:14.748169 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:14.748270 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:14.749155 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:14.749159 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:14.749162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:14.749176 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:14.749205 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:14.749218 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:14.749298 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:14.749504 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 06:06:14.749563 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 06:06:14.749995 24316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 06:06:14.750188 24316 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:14.750259 24316 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:14.750360 24316 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:14.751420 24316 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 06:06:14.751744 24316 leader_election.cc:302] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2eb4735e89614d12aa568c20f6f4ccc6; no voters:
I20260620 06:06:14.752019 24316 leader_election.cc:288] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:14.752177 24326 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:14.752441 24326 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [term 1 LEADER]: Becoming Leader. State: Replica: 2eb4735e89614d12aa568c20f6f4ccc6, State: Running, Role: LEADER
I20260620 06:06:14.752787 24326 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } }
I20260620 06:06:14.753351 24316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:14.754448 24328 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2eb4735e89614d12aa568c20f6f4ccc6. Latest consensus state: current_term: 1 leader_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } } }
I20260620 06:06:14.754571 24328 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:14.754966 24335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:14.754980 24327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2eb4735e89614d12aa568c20f6f4ccc6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35853 } } }
I20260620 06:06:14.755722 24327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:14.758450 24335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:14.764900 24335 catalog_manager.cc:1347] Generated new cluster ID: 73b140fde8204bbd885200dbfe0c5ef1
I20260620 06:06:14.764994 24335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:14.782013 24335 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:14.783031 24335 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:14.794682 24335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2eb4735e89614d12aa568c20f6f4ccc6: Generated new TSK 0
I20260620 06:06:14.795465 24335 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:14.802000 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24248
2026-06-20T06:06:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:14 dist-test-slave-4h14 krb5kdc[24347](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:14 dist-test-slave-4h14 krb5kdc[24347](info): set up 2 sockets
Jun 20 06:06:14 dist-test-slave-4h14 krb5kdc[24347](info): commencing operation
krb5kdc: starting...
W20260620 06:06:15.841014 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:15 dist-test-slave-4h14 krb5kdc[24347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:15.973150 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:16.090287 24360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:16.090569 24360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:16.090651 24360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:16.094919 24360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:16.095008 24360 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:16.095028 24360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:16.095045 24360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:16.095103 24360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:16.100988 24360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24360
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:16.102187 24360 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:16.103318 24360 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:16.112375 24366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:16.112435 24365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:16.112375 24368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:16.112797 24360 server_base.cc:1034] running on GCE node
I20260620 06:06:16.113511 24360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:16.114763 24360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:16.115978 24360 hybrid_clock.cc:648] HybridClock initialized: now 1781935576115945 us; error 52 us; skew 500 ppm
Jun 20 06:06:16 dist-test-slave-4h14 krb5kdc[24347](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935576, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:16.119169 24360 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:16.120249 24360 webserver.cc:460] Webserver started at http://127.0.0.1:41661/ using document root <none> and password file <none>
I20260620 06:06:16.120981 24360 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:16.121085 24360 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:16.121335 24360 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:16.123556 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "0596f8d263f04e3485cb3d93545aa4e9"
format_stamp: "Formatted at 2026-06-20 06:06:16 on dist-test-slave-4h14"
server_key: "9b30e2ed0b9493457e34e3be00504c75"
server_key_iv: "c7d6d32b6ffcd5b4e3fa9626f5d46389"
server_key_version: "encryptionkey@0"
I20260620 06:06:16.124181 24360 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "0596f8d263f04e3485cb3d93545aa4e9"
format_stamp: "Formatted at 2026-06-20 06:06:16 on dist-test-slave-4h14"
server_key: "9b30e2ed0b9493457e34e3be00504c75"
server_key_iv: "c7d6d32b6ffcd5b4e3fa9626f5d46389"
server_key_version: "encryptionkey@0"
I20260620 06:06:16.128229 24360 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:16.131060 24375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:16.132426 24360 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:16.132591 24360 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "0596f8d263f04e3485cb3d93545aa4e9"
format_stamp: "Formatted at 2026-06-20 06:06:16 on dist-test-slave-4h14"
server_key: "9b30e2ed0b9493457e34e3be00504c75"
server_key_iv: "c7d6d32b6ffcd5b4e3fa9626f5d46389"
server_key_version: "encryptionkey@0"
I20260620 06:06:16.132772 24360 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:16.137852 24360 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:16.141431 24360 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:16.141644 24360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:16.149107 24360 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36845
I20260620 06:06:16.149134 24427 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36845 every 8 connection(s)
I20260620 06:06:16.150210 24360 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:16.153342 24428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:16.159140 24428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9: Bootstrap starting.
I20260620 06:06:16.160025 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24360
I20260620 06:06:16.160176 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:16.160526 3532 external_mini_cluster.cc:1428] Setting key b11ac8c721beb96f541ec9942a7a665f
I20260620 06:06:16.162328 24428 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:16.163092 24428 log.cc:824] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:16.165581 24428 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9: No bootstrap required, opened a new log
I20260620 06:06:16.166267 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:16.161829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42959 (local address 127.0.0.1:36845)
0620 06:06:16.162338 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:16.162354 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:16.162392 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:06:16.163135 (+ 743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:16.163154 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:16.163568 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:16.163980 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:16.164896 (+ 916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:16.164901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:16.164914 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:16.164926 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:16.164982 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:16.165002 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:16.165091 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:16.165234 (+ 143us) server_negotiation.cc:299] Negotiation successful
0620 06:06:16.165421 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":152,"threads_started":1}
I20260620 06:06:16.169371 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:16.170008 24428 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0596f8d263f04e3485cb3d93545aa4e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260620 06:06:16.170277 24428 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:16.170356 24428 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0596f8d263f04e3485cb3d93545aa4e9, State: Initialized, Role: FOLLOWER
I20260620 06:06:16.170856 24428 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0596f8d263f04e3485cb3d93545aa4e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260620 06:06:16.171012 24428 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:16.171098 24428 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:16.171200 24428 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:16.172602 24428 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0596f8d263f04e3485cb3d93545aa4e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260620 06:06:16.173054 24428 leader_election.cc:302] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0596f8d263f04e3485cb3d93545aa4e9; no voters:
I20260620 06:06:16.173429 24428 leader_election.cc:288] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:16.173564 24431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:16.171619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45393 (local address 127.0.0.1:36845)
0620 06:06:16.171758 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:16.171763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:16.171928 (+ 165us) server_negotiation.cc:407] Connection header received
0620 06:06:16.172062 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:16.172064 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:16.172098 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:16.172167 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:16.173072 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:16.173077 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:16.173081 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:16.173094 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:16.173124 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:16.173138 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:16.173245 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:16.173396 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 06:06:16.173441 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:06:16.173588 24438 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:16.174037 24438 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [term 1 LEADER]: Becoming Leader. State: Replica: 0596f8d263f04e3485cb3d93545aa4e9, State: Running, Role: LEADER
I20260620 06:06:16.174425 24438 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0596f8d263f04e3485cb3d93545aa4e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } }
I20260620 06:06:16.174903 24428 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:16.176409 24439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0596f8d263f04e3485cb3d93545aa4e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0596f8d263f04e3485cb3d93545aa4e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260620 06:06:16.176429 24440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0596f8d263f04e3485cb3d93545aa4e9. Latest consensus state: current_term: 1 leader_uuid: "0596f8d263f04e3485cb3d93545aa4e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0596f8d263f04e3485cb3d93545aa4e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36845 } } }
I20260620 06:06:16.176563 24439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:16.176566 24440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:16.176956 24448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:16.180222 24448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:16.186698 24448 catalog_manager.cc:1347] Generated new cluster ID: 50ffefb74e124027843dd5b404ce7908
I20260620 06:06:16.186802 24448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:16.205332 24448 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:16.205708 24448 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:16.209992 24448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0596f8d263f04e3485cb3d93545aa4e9: Generated new TSK 0
I20260620 06:06:16.210790 24448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:16.220928 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24360
2026-06-20T06:06:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:16 dist-test-slave-4h14 krb5kdc[24459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:16 dist-test-slave-4h14 krb5kdc[24459](info): set up 2 sockets
Jun 20 06:06:16 dist-test-slave-4h14 krb5kdc[24459](info): commencing operation
krb5kdc: starting...
W20260620 06:06:17.269265 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:17 dist-test-slave-4h14 krb5kdc[24459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:17.400247 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:17.523134 24472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:17.523451 24472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:17.523497 24472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:17.527789 24472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:17.527874 24472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:17.527895 24472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:17.527913 24472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:17.527928 24472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:17.533644 24472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:17.534808 24472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:17.535878 24472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:17.545058 24480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:17.545079 24478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:17.545068 24477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:17.545449 24472 server_base.cc:1034] running on GCE node
I20260620 06:06:17.546172 24472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:17.547448 24472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:17.548621 24472 hybrid_clock.cc:648] HybridClock initialized: now 1781935577548608 us; error 30 us; skew 500 ppm
Jun 20 06:06:17 dist-test-slave-4h14 krb5kdc[24459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:17.555140 24472 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:17.556185 24472 webserver.cc:460] Webserver started at http://127.0.0.1:45863/ using document root <none> and password file <none>
I20260620 06:06:17.556891 24472 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:17.556960 24472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:17.557138 24472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:17.559283 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "ac8d4bacd4b441829cdf45294d39b919"
format_stamp: "Formatted at 2026-06-20 06:06:17 on dist-test-slave-4h14"
server_key: "5d3e46e93ae5728c3b3f9094a8c76fa5"
server_key_iv: "85e92a7971e8209d6adcfce40d4749cf"
server_key_version: "encryptionkey@0"
I20260620 06:06:17.559834 24472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "ac8d4bacd4b441829cdf45294d39b919"
format_stamp: "Formatted at 2026-06-20 06:06:17 on dist-test-slave-4h14"
server_key: "5d3e46e93ae5728c3b3f9094a8c76fa5"
server_key_iv: "85e92a7971e8209d6adcfce40d4749cf"
server_key_version: "encryptionkey@0"
I20260620 06:06:17.563804 24472 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:06:17.566463 24487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:17.567745 24472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:06:17.567880 24472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "ac8d4bacd4b441829cdf45294d39b919"
format_stamp: "Formatted at 2026-06-20 06:06:17 on dist-test-slave-4h14"
server_key: "5d3e46e93ae5728c3b3f9094a8c76fa5"
server_key_iv: "85e92a7971e8209d6adcfce40d4749cf"
server_key_version: "encryptionkey@0"
I20260620 06:06:17.567999 24472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:17.596406 24472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:17.599999 24472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:17.600258 24472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:17.607798 24472 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37325
I20260620 06:06:17.607806 24539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37325 every 8 connection(s)
I20260620 06:06:17.608924 24472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:17.611784 24540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:17.617586 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24472
I20260620 06:06:17.617779 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:17.618101 3532 external_mini_cluster.cc:1428] Setting key 77146cc310cf58a61115babe82ed458f
I20260620 06:06:17.618366 24540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919: Bootstrap starting.
I20260620 06:06:17.621259 24540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:17.622080 24540 log.cc:824] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:17.624105 24540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919: No bootstrap required, opened a new log
Jun 20 06:06:17 dist-test-slave-4h14 krb5kdc[24459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935577, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:06:17.628113 24540 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d4bacd4b441829cdf45294d39b919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260620 06:06:17.628393 24540 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:17.628473 24540 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac8d4bacd4b441829cdf45294d39b919, State: Initialized, Role: FOLLOWER
I20260620 06:06:17.629060 24540 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d4bacd4b441829cdf45294d39b919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260620 06:06:17.629247 24540 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:17.629336 24540 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:17.629446 24540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:17.630709 24540 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d4bacd4b441829cdf45294d39b919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260620 06:06:17.631125 24540 leader_election.cc:302] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac8d4bacd4b441829cdf45294d39b919; no voters:
I20260620 06:06:17.631546 24540 leader_election.cc:288] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:17.631657 24545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:17.631955 24545 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [term 1 LEADER]: Becoming Leader. State: Replica: ac8d4bacd4b441829cdf45294d39b919, State: Running, Role: LEADER
I20260620 06:06:17.632350 24545 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d4bacd4b441829cdf45294d39b919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } }
I20260620 06:06:17.633090 24540 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:17.634186 24547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac8d4bacd4b441829cdf45294d39b919. Latest consensus state: current_term: 1 leader_uuid: "ac8d4bacd4b441829cdf45294d39b919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d4bacd4b441829cdf45294d39b919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260620 06:06:17.634315 24547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:17.634346 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:17.619567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:46997 (local address 127.0.0.1:37325)
0620 06:06:17.620091 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:17.620107 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:17.620147 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:06:17.620957 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:17.620977 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:17.621367 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:17.621782 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:17.622618 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:17.623779 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:17.624828 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:17.625326 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:17.625504 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:17.628133 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:17.628138 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:17.628153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:17.628184 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:17.630498 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:17.631152 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:17.631159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:17.631165 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:17.631315 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:17.631719 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:17.631724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:17.631726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:17.632297 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:17.632563 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:17.632956 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 06:06:17.633214 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":173,"threads_started":1}
I20260620 06:06:17.634907 24546 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac8d4bacd4b441829cdf45294d39b919" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac8d4bacd4b441829cdf45294d39b919" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37325 } } }
I20260620 06:06:17.635051 24546 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919 [sys.catalog]: This master's current role is: LEADER
W20260620 06:06:17.638238 24560 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:17.638347 24560 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:17.638446 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:17.638469 24554 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:17.639386 24554 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:17.645844 24554 catalog_manager.cc:1347] Generated new cluster ID: 5de90ce37db64615a09c6044daa5a2d9
I20260620 06:06:17.645920 24554 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:17.649398 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:17.640757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:33641 (local address 127.0.0.1:37325)
0620 06:06:17.640894 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:17.640899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:17.640911 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:17.640946 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:17.640949 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:17.640995 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:17.641081 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:17.641844 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:17.642706 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:17.643523 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:17.643784 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:17.643907 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:17.645438 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:17.645442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:17.645445 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:17.645472 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:17.647297 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:17.647891 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:17.647894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:17.647896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:17.647975 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:17.648327 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:17.648330 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:17.648331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:17.648522 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:17.648649 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:17.648977 (+ 328us) server_negotiation.cc:299] Negotiation successful
0620 06:06:17.649069 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:06:17.676203 24554 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:17.677310 24554 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:17.687232 24554 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac8d4bacd4b441829cdf45294d39b919: Generated new TSK 0
I20260620 06:06:17.688037 24554 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:17.696571 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24472
W20260620 06:06:17.702148 24457 connection.cc:535] client connection to 127.0.0.1:37325 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37325): Connection reset by peer (error 104)
2026-06-20T06:06:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1475 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:17 dist-test-slave-4h14 krb5kdc[24571](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:17 dist-test-slave-4h14 krb5kdc[24571](info): set up 2 sockets
Jun 20 06:06:17 dist-test-slave-4h14 krb5kdc[24571](info): commencing operation
krb5kdc: starting...
W20260620 06:06:18.737078 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:18 dist-test-slave-4h14 krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:18.864949 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:18.982199 24584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:18.982501 24584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:18.982579 24584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:18.986876 24584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:18.986961 24584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:18.986981 24584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:18.986999 24584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:18.987015 24584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:18.993034 24584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40291
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:18.994378 24584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:18.995728 24584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:19.006768 24590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:19.006721 24592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:19.006686 24589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:19.007153 24584 server_base.cc:1034] running on GCE node
I20260620 06:06:19.007974 24584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:19.009289 24584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:19.010486 24584 hybrid_clock.cc:648] HybridClock initialized: now 1781935579010450 us; error 56 us; skew 500 ppm
Jun 20 06:06:19 dist-test-slave-4h14 krb5kdc[24571](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935579, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:19.013494 24584 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:19.014565 24584 webserver.cc:460] Webserver started at http://127.0.0.1:35877/ using document root <none> and password file <none>
I20260620 06:06:19.015194 24584 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:19.015290 24584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:19.015581 24584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:19.017920 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "0c32a426d4d34633bc5f29e339c51849"
format_stamp: "Formatted at 2026-06-20 06:06:19 on dist-test-slave-4h14"
server_key: "cfbaad9af32bb818d1ec2f61df33f3ff"
server_key_iv: "2c0ad63e41a3ed7d193b925c493d56ba"
server_key_version: "encryptionkey@0"
I20260620 06:06:19.018541 24584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "0c32a426d4d34633bc5f29e339c51849"
format_stamp: "Formatted at 2026-06-20 06:06:19 on dist-test-slave-4h14"
server_key: "cfbaad9af32bb818d1ec2f61df33f3ff"
server_key_iv: "2c0ad63e41a3ed7d193b925c493d56ba"
server_key_version: "encryptionkey@0"
I20260620 06:06:19.022776 24584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:19.025477 24599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:19.026712 24584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260620 06:06:19.026873 24584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "0c32a426d4d34633bc5f29e339c51849"
format_stamp: "Formatted at 2026-06-20 06:06:19 on dist-test-slave-4h14"
server_key: "cfbaad9af32bb818d1ec2f61df33f3ff"
server_key_iv: "2c0ad63e41a3ed7d193b925c493d56ba"
server_key_version: "encryptionkey@0"
I20260620 06:06:19.027004 24584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:19.041395 24584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:19.044587 24584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:19.044867 24584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:19.052541 24584 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40291
I20260620 06:06:19.052563 24651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40291 every 8 connection(s)
I20260620 06:06:19.053638 24584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:19.056632 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:19.062085 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24584
I20260620 06:06:19.062289 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:19.062589 3532 external_mini_cluster.cc:1428] Setting key e59087b0d9019232fbc6054bf519d9d5
I20260620 06:06:19.062942 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849: Bootstrap starting.
I20260620 06:06:19.065795 24652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:19.066581 24652 log.cc:824] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:19.068614 24652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849: No bootstrap required, opened a new log
I20260620 06:06:19.072739 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:19.064111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:37527 (local address 127.0.0.1:40291)
0620 06:06:19.064754 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:19.064786 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:19.064828 (+ 42us) server_negotiation.cc:407] Connection header received
0620 06:06:19.065540 (+ 712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:19.065558 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:19.065990 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:19.066369 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:19.066857 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:19.067849 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:19.068883 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:19.069378 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:19.069556 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:19.070601 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:19.070606 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:19.070621 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:19.071111 (+ 490us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:19.071170 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:19.071195 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:19.071347 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:19.071650 (+ 303us) server_negotiation.cc:299] Negotiation successful
0620 06:06:19.071829 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":161,"threads_started":1}
I20260620 06:06:19.073010 24652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c32a426d4d34633bc5f29e339c51849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260620 06:06:19.073361 24652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:19.073418 24652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c32a426d4d34633bc5f29e339c51849, State: Initialized, Role: FOLLOWER
I20260620 06:06:19.073925 24652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c32a426d4d34633bc5f29e339c51849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260620 06:06:19.074091 24652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:19.074183 24652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:19.074285 24652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:19.075500 24652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c32a426d4d34633bc5f29e339c51849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260620 06:06:19.075894 24652 leader_election.cc:302] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c32a426d4d34633bc5f29e339c51849; no voters:
I20260620 06:06:19.076280 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:19.076311 24652 leader_election.cc:288] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:19.076381 24657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:19.076597 24657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [term 1 LEADER]: Becoming Leader. State: Replica: 0c32a426d4d34633bc5f29e339c51849, State: Running, Role: LEADER
I20260620 06:06:19.076956 24657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c32a426d4d34633bc5f29e339c51849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } }
I20260620 06:06:19.077741 24652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:19.078747 24658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c32a426d4d34633bc5f29e339c51849. Latest consensus state: current_term: 1 leader_uuid: "0c32a426d4d34633bc5f29e339c51849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c32a426d4d34633bc5f29e339c51849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } } }
I20260620 06:06:19.078879 24658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:19.079286 24671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:19.079344 24659 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c32a426d4d34633bc5f29e339c51849" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c32a426d4d34633bc5f29e339c51849" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40291 } } }
I20260620 06:06:19.079460 24659 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:19.083006 24671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:19.084976 24655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:19.078769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:40905 (local address 127.0.0.1:40291)
0620 06:06:19.079173 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:19.079178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:19.080056 (+ 878us) server_negotiation.cc:407] Connection header received
0620 06:06:19.080269 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:19.080275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:19.080334 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:19.080445 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:19.080789 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:19.081568 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:19.082697 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:19.083175 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:19.083319 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:19.084357 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:19.084360 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:19.084366 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:19.084380 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:19.084408 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:19.084418 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:19.084548 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:19.084784 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 06:06:19.084822 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326}
I20260620 06:06:19.091601 24671 catalog_manager.cc:1347] Generated new cluster ID: acae400db5244423a538202ad938b30e
I20260620 06:06:19.091704 24671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:19.107052 24671 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:19.108431 24671 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:19.119366 24671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c32a426d4d34633bc5f29e339c51849: Generated new TSK 0
I20260620 06:06:19.120316 24671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:19.124872 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24584
W20260620 06:06:19.130486 24569 connection.cc:535] client connection to 127.0.0.1:40291 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40291): Connection reset by peer (error 104)
2026-06-20T06:06:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1428 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:19 dist-test-slave-4h14 krb5kdc[24683](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:19 dist-test-slave-4h14 krb5kdc[24683](info): set up 2 sockets
Jun 20 06:06:19 dist-test-slave-4h14 krb5kdc[24683](info): commencing operation
krb5kdc: starting...
W20260620 06:06:20.185137 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:20 dist-test-slave-4h14 krb5kdc[24683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:20.312254 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37855
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37855
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:20.429474 24696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:20.429836 24696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:20.429912 24696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:20.434194 24696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:20.434304 24696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:20.434343 24696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:20.434404 24696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:20.434443 24696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:20.440199 24696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37855
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:20.441434 24696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:20.442597 24696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:20.451630 24701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:20.451630 24704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:20.451635 24702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:20.451860 24696 server_base.cc:1034] running on GCE node
I20260620 06:06:20.452646 24696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:20.453998 24696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:20.455185 24696 hybrid_clock.cc:648] HybridClock initialized: now 1781935580455151 us; error 52 us; skew 500 ppm
Jun 20 06:06:20 dist-test-slave-4h14 krb5kdc[24683](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:20.458223 24696 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:20.459241 24696 webserver.cc:460] Webserver started at http://127.0.0.1:33131/ using document root <none> and password file <none>
I20260620 06:06:20.459894 24696 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:20.459995 24696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:20.460232 24696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:20.462462 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "7041a4011a49493e85c47d6095e9768b"
format_stamp: "Formatted at 2026-06-20 06:06:20 on dist-test-slave-4h14"
server_key: "baa95658b16592ff28890910f531b6dc"
server_key_iv: "931e54caef6a746e662b4a1c7d2a5687"
server_key_version: "encryptionkey@0"
I20260620 06:06:20.463049 24696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "7041a4011a49493e85c47d6095e9768b"
format_stamp: "Formatted at 2026-06-20 06:06:20 on dist-test-slave-4h14"
server_key: "baa95658b16592ff28890910f531b6dc"
server_key_iv: "931e54caef6a746e662b4a1c7d2a5687"
server_key_version: "encryptionkey@0"
I20260620 06:06:20.467167 24696 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260620 06:06:20.469990 24711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:20.471374 24696 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:20.471531 24696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "7041a4011a49493e85c47d6095e9768b"
format_stamp: "Formatted at 2026-06-20 06:06:20 on dist-test-slave-4h14"
server_key: "baa95658b16592ff28890910f531b6dc"
server_key_iv: "931e54caef6a746e662b4a1c7d2a5687"
server_key_version: "encryptionkey@0"
I20260620 06:06:20.471662 24696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:20.477010 24696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:20.480610 24696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:20.480918 24696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:20.488482 24696 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37855
I20260620 06:06:20.488500 24763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37855 every 8 connection(s)
I20260620 06:06:20.489538 24696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:20.492507 24764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:20.498719 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b: Bootstrap starting.
I20260620 06:06:20.499264 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24696
I20260620 06:06:20.499413 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:20.499737 3532 external_mini_cluster.cc:1428] Setting key 90837c729b4fb8d502a3233adf1b9cf6
I20260620 06:06:20.501729 24764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:20.502715 24764 log.cc:824] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:20.505287 24764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b: No bootstrap required, opened a new log
I20260620 06:06:20.506073 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:20.501232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:54835 (local address 127.0.0.1:37855)
0620 06:06:20.501727 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:20.501748 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:20.501797 (+ 49us) server_negotiation.cc:407] Connection header received
0620 06:06:20.502781 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:20.502808 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:20.503225 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:20.503538 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:20.504424 (+ 886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:20.504429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:20.504444 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:20.504457 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:20.504522 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:20.504548 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:20.504725 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:20.504945 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 06:06:20.505113 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":127,"threads_started":1}
I20260620 06:06:20.509187 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:20.509814 24764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041a4011a49493e85c47d6095e9768b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37855 } }
I20260620 06:06:20.510043 24764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:20.510103 24764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7041a4011a49493e85c47d6095e9768b, State: Initialized, Role: FOLLOWER
I20260620 06:06:20.510594 24764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041a4011a49493e85c47d6095e9768b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37855 } }
I20260620 06:06:20.510722 24764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:20.510809 24764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:20.510922 24764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:20.512274 24764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041a4011a49493e85c47d6095e9768b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37855 } }
I20260620 06:06:20.512702 24764 leader_election.cc:302] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7041a4011a49493e85c47d6095e9768b; no voters:
I20260620 06:06:20.513087 24764 leader_election.cc:288] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:20.513201 24774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:20.513366 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:20.511377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:43523 (local address 127.0.0.1:37855)
0620 06:06:20.511518 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:20.511523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:20.511760 (+ 237us) server_negotiation.cc:407] Connection header received
0620 06:06:20.511878 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:20.511880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:20.511914 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:20.511981 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:20.512830 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:20.512832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:20.512834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:20.512862 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:20.512920 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:20.512934 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:20.513019 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:20.513165 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 06:06:20.513210 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:06:20.513439 24774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [term 1 LEADER]: Becoming Leader. State: Replica: 7041a4011a49493e85c47d6095e9768b, State: Running, Role: LEADER
I20260620 06:06:20.513767 24774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041a4011a49493e85c47d6095e9768b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37855 } }
I20260620 06:06:20.514686 24764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:20.515043 24776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7041a4011a49493e85c47d6095e9768b. Latest consensus state: current_term: 1 leader_uuid: "7041a4011a49493e85c47d6095e9768b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041a4011a49493e85c47d6095e9768b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37855 } } }
I20260620 06:06:20.515165 24776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:20.515472 24775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7041a4011a49493e85c47d6095e9768b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7041a4011a49493e85c47d6095e9768b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37855 } } }
I20260620 06:06:20.515583 24775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:20.515851 24778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:20.519090 24778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:20.524796 24778 catalog_manager.cc:1347] Generated new cluster ID: 037e389cc12f492e9abdd28cd9753f6f
I20260620 06:06:20.524875 24778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:20.533865 24778 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:20.534279 24778 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:20.541688 24778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7041a4011a49493e85c47d6095e9768b: Generated new TSK 0
I20260620 06:06:20.542533 24778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:20.547863 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24696
2026-06-20T06:06:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1423 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:20 dist-test-slave-4h14 krb5kdc[24795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:20 dist-test-slave-4h14 krb5kdc[24795](info): set up 2 sockets
Jun 20 06:06:20 dist-test-slave-4h14 krb5kdc[24795](info): commencing operation
krb5kdc: starting...
W20260620 06:06:21.593351 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:21 dist-test-slave-4h14 krb5kdc[24795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:21.720268 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:21.835582 24808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:21.835927 24808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:21.836012 24808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:21.840137 24808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:21.840243 24808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:21.840281 24808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:21.840320 24808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:21.840358 24808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:21.845831 24808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:21.847048 24808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:21.848201 24808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:21.857115 24813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:21.857223 24808 server_base.cc:1034] running on GCE node
W20260620 06:06:21.857122 24814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:21.857125 24816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:21.858037 24808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:21.859232 24808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:21.860416 24808 hybrid_clock.cc:648] HybridClock initialized: now 1781935581860385 us; error 43 us; skew 500 ppm
Jun 20 06:06:21 dist-test-slave-4h14 krb5kdc[24795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935581, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:21.863443 24808 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:21.864444 24808 webserver.cc:460] Webserver started at http://127.0.0.1:38651/ using document root <none> and password file <none>
I20260620 06:06:21.865163 24808 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:21.865259 24808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:21.865489 24808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:21.867667 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "284388c0674b4f1eb29fd1de56883bc8"
format_stamp: "Formatted at 2026-06-20 06:06:21 on dist-test-slave-4h14"
server_key: "d57a0bb0b018924d88052e75b8762973"
server_key_iv: "397bed00cc9930eb98f586326c90f432"
server_key_version: "encryptionkey@0"
I20260620 06:06:21.868242 24808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "284388c0674b4f1eb29fd1de56883bc8"
format_stamp: "Formatted at 2026-06-20 06:06:21 on dist-test-slave-4h14"
server_key: "d57a0bb0b018924d88052e75b8762973"
server_key_iv: "397bed00cc9930eb98f586326c90f432"
server_key_version: "encryptionkey@0"
I20260620 06:06:21.872200 24808 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:21.874912 24823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:21.876200 24808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260620 06:06:21.876359 24808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "284388c0674b4f1eb29fd1de56883bc8"
format_stamp: "Formatted at 2026-06-20 06:06:21 on dist-test-slave-4h14"
server_key: "d57a0bb0b018924d88052e75b8762973"
server_key_iv: "397bed00cc9930eb98f586326c90f432"
server_key_version: "encryptionkey@0"
I20260620 06:06:21.876503 24808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:21.896507 24808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:21.900044 24808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:21.900225 24808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:21.907835 24808 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40587
I20260620 06:06:21.907860 24875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40587 every 8 connection(s)
I20260620 06:06:21.908921 24808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:21.911880 24876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:21.917160 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24808
I20260620 06:06:21.917287 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:21.917580 3532 external_mini_cluster.cc:1428] Setting key ff50219a9a32b867a22f045f925c0359
I20260620 06:06:21.917896 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8: Bootstrap starting.
I20260620 06:06:21.920522 24876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:21.921332 24876 log.cc:824] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:21.923297 24876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8: No bootstrap required, opened a new log
Jun 20 06:06:21 dist-test-slave-4h14 krb5kdc[24795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 06:06:21.927532 24876 raft_consensus.cc:348] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284388c0674b4f1eb29fd1de56883bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260620 06:06:21.927811 24876 raft_consensus.cc:374] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:21.927872 24876 raft_consensus.cc:729] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 284388c0674b4f1eb29fd1de56883bc8, State: Initialized, Role: FOLLOWER
I20260620 06:06:21.928349 24876 consensus_queue.cc:260] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284388c0674b4f1eb29fd1de56883bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260620 06:06:21.928546 24876 raft_consensus.cc:388] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:21.928613 24876 raft_consensus.cc:482] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:21.928795 24876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:21.929718 24876 raft_consensus.cc:504] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284388c0674b4f1eb29fd1de56883bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260620 06:06:21.930069 24876 leader_election.cc:302] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 284388c0674b4f1eb29fd1de56883bc8; no voters:
I20260620 06:06:21.930461 24876 leader_election.cc:288] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:21.930529 24881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:21.930750 24881 raft_consensus.cc:686] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [term 1 LEADER]: Becoming Leader. State: Replica: 284388c0674b4f1eb29fd1de56883bc8, State: Running, Role: LEADER
I20260620 06:06:21.931115 24881 consensus_queue.cc:237] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284388c0674b4f1eb29fd1de56883bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } }
I20260620 06:06:21.931877 24876 sys_catalog.cc:564] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:21.932767 24883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 284388c0674b4f1eb29fd1de56883bc8. Latest consensus state: current_term: 1 leader_uuid: "284388c0674b4f1eb29fd1de56883bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284388c0674b4f1eb29fd1de56883bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } } }
I20260620 06:06:21.932905 24883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:21.933264 24882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "284388c0674b4f1eb29fd1de56883bc8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "284388c0674b4f1eb29fd1de56883bc8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40587 } } }
I20260620 06:06:21.933308 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:21.918973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:52907 (local address 127.0.0.1:40587)
0620 06:06:21.919447 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:21.919462 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:21.919498 (+ 36us) server_negotiation.cc:407] Connection header received
0620 06:06:21.920137 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:21.920156 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:21.920519 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:21.920934 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:21.921775 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:21.922762 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:21.923726 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:21.924208 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:21.924375 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:21.927417 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:21.927423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:21.927438 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:21.927469 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:21.929499 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:21.930119 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:21.930126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:21.930132 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:21.930317 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:21.930696 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:21.930699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:21.930701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:21.931272 (+ 571us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:21.931495 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:21.931836 (+ 341us) server_negotiation.cc:299] Negotiation successful
0620 06:06:21.932279 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":165,"threads_started":1}
I20260620 06:06:21.933348 24882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:21.936882 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260620 06:06:21.937126 24897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:21.937211 24897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:21.937264 24890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:21.938248 24890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:21.945092 24890 catalog_manager.cc:1347] Generated new cluster ID: cdcf724eee464dd889e7f8d10ea577ae
I20260620 06:06:21.945175 24890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:21.948591 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:21.939197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:36853 (local address 127.0.0.1:40587)
0620 06:06:21.939302 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:21.939308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:21.939494 (+ 186us) server_negotiation.cc:407] Connection header received
0620 06:06:21.939679 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:21.939682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:21.939735 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:21.939787 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:21.940789 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:21.941646 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:21.942494 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:21.942827 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:21.942949 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:21.944868 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:21.944872 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:21.944875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:21.944903 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:21.946688 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:21.947244 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:21.947248 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:21.947250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:21.947341 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:21.947665 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:21.947668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:21.947670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:21.947881 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:21.948057 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:21.948316 (+ 259us) server_negotiation.cc:299] Negotiation successful
0620 06:06:21.948419 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 06:06:21.960577 24890 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:21.962005 24890 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:21.967116 24890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 284388c0674b4f1eb29fd1de56883bc8: Generated new TSK 0
I20260620 06:06:21.967986 24890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:21.980086 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24808
W20260620 06:06:21.985567 24793 connection.cc:535] client connection to 127.0.0.1:40587 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40587): Connection reset by peer (error 104)
2026-06-20T06:06:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1431 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:22 dist-test-slave-4h14 krb5kdc[24907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:22 dist-test-slave-4h14 krb5kdc[24907](info): set up 2 sockets
Jun 20 06:06:22 dist-test-slave-4h14 krb5kdc[24907](info): commencing operation
krb5kdc: starting...
W20260620 06:06:23.020947 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:23 dist-test-slave-4h14 krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:23.141873 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35255
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:23.254381 24920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:23.254694 24920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:23.254746 24920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:23.258869 24920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:23.258941 24920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:23.258961 24920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:23.258978 24920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:23.258993 24920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:23.264714 24920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35255
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:23.265899 24920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:23.266928 24920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:23.276001 24926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:23.275998 24928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:23.275998 24925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:23.276435 24920 server_base.cc:1034] running on GCE node
I20260620 06:06:23.277078 24920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:23.278191 24920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:23.279384 24920 hybrid_clock.cc:648] HybridClock initialized: now 1781935583279353 us; error 49 us; skew 500 ppm
Jun 20 06:06:23 dist-test-slave-4h14 krb5kdc[24907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:23.286187 24920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:23.287424 24920 webserver.cc:460] Webserver started at http://127.0.0.1:44975/ using document root <none> and password file <none>
I20260620 06:06:23.288236 24920 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:23.288335 24920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:23.288599 24920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:23.291610 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "dbe766169cd44d9c89afa69edd1c17ab"
format_stamp: "Formatted at 2026-06-20 06:06:23 on dist-test-slave-4h14"
server_key: "fd8b99426ae3ed6b6a490d5e5afcc500"
server_key_iv: "a8ed901ac662b7dbffbc356a2891ae1c"
server_key_version: "encryptionkey@0"
I20260620 06:06:23.292275 24920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "dbe766169cd44d9c89afa69edd1c17ab"
format_stamp: "Formatted at 2026-06-20 06:06:23 on dist-test-slave-4h14"
server_key: "fd8b99426ae3ed6b6a490d5e5afcc500"
server_key_iv: "a8ed901ac662b7dbffbc356a2891ae1c"
server_key_version: "encryptionkey@0"
I20260620 06:06:23.297807 24920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260620 06:06:23.300922 24935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:23.302345 24920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:23.302467 24920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "dbe766169cd44d9c89afa69edd1c17ab"
format_stamp: "Formatted at 2026-06-20 06:06:23 on dist-test-slave-4h14"
server_key: "fd8b99426ae3ed6b6a490d5e5afcc500"
server_key_iv: "a8ed901ac662b7dbffbc356a2891ae1c"
server_key_version: "encryptionkey@0"
I20260620 06:06:23.302575 24920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:23.313809 24920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:23.317060 24920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:23.317234 24920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:23.325109 24920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35255
I20260620 06:06:23.325134 24987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35255 every 8 connection(s)
I20260620 06:06:23.326148 24920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:23.328512 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 24920
I20260620 06:06:23.328694 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:23.328996 3532 external_mini_cluster.cc:1428] Setting key d7a1b36840c9c7414063277470d6ef2a
I20260620 06:06:23.329458 24988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:23.336090 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab: Bootstrap starting.
I20260620 06:06:23.338485 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:23.330514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:56531 (local address 127.0.0.1:35255)
0620 06:06:23.331021 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:23.331037 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:23.331077 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:06:23.331817 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:23.331835 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:23.332224 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:23.332606 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:23.333064 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:23.334089 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:23.335004 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:23.335511 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:23.335707 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:23.336757 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:23.336763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:23.336777 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:23.337211 (+ 434us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:23.337270 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:23.337293 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:23.337433 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:23.337587 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 06:06:23.337736 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":169,"threads_started":1}
I20260620 06:06:23.338727 24988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:23.339704 24988 log.cc:824] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:23.341732 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:23.342003 24988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab: No bootstrap required, opened a new log
I20260620 06:06:23.346555 24988 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe766169cd44d9c89afa69edd1c17ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260620 06:06:23.346911 24988 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:23.346988 24988 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe766169cd44d9c89afa69edd1c17ab, State: Initialized, Role: FOLLOWER
I20260620 06:06:23.347595 24988 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe766169cd44d9c89afa69edd1c17ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260620 06:06:23.347783 24988 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:23.347867 24988 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:23.348007 24988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:23.348341 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:23.343748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:38679 (local address 127.0.0.1:35255)
0620 06:06:23.343891 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:23.343898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:23.344119 (+ 221us) server_negotiation.cc:407] Connection header received
0620 06:06:23.344233 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:23.344238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:23.344303 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:23.344405 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:23.344696 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:23.345477 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:23.346343 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:23.346678 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:23.346799 (+ 121us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:23.347787 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:23.347793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:23.347799 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:23.347817 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:23.347848 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:23.347863 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:23.347961 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:23.348129 (+ 168us) server_negotiation.cc:299] Negotiation successful
0620 06:06:23.348174 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:23.349427 24988 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe766169cd44d9c89afa69edd1c17ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260620 06:06:23.349870 24988 leader_election.cc:302] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe766169cd44d9c89afa69edd1c17ab; no voters:
I20260620 06:06:23.350260 24988 leader_election.cc:288] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:23.350373 24998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:23.350653 24998 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [term 1 LEADER]: Becoming Leader. State: Replica: dbe766169cd44d9c89afa69edd1c17ab, State: Running, Role: LEADER
I20260620 06:06:23.351046 24998 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe766169cd44d9c89afa69edd1c17ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } }
I20260620 06:06:23.351619 24988 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:23.352824 25000 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe766169cd44d9c89afa69edd1c17ab. Latest consensus state: current_term: 1 leader_uuid: "dbe766169cd44d9c89afa69edd1c17ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe766169cd44d9c89afa69edd1c17ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } } }
I20260620 06:06:23.352962 25000 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:23.353325 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe766169cd44d9c89afa69edd1c17ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe766169cd44d9c89afa69edd1c17ab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35255 } } }
I20260620 06:06:23.353477 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab [sys.catalog]: This master's current role is: LEADER
W20260620 06:06:23.356851 25013 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:23.356920 25013 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:23.356980 25007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:23.357710 25007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:23.362943 25007 catalog_manager.cc:1347] Generated new cluster ID: d7a1070e5d7743cf89ed670ed6b6392e
I20260620 06:06:23.363024 25007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:23.387730 25007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:23.388871 25007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:23.397594 25007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbe766169cd44d9c89afa69edd1c17ab: Generated new TSK 0
I20260620 06:06:23.398427 25007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:23.410565 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 24920
W20260620 06:06:23.416285 24905 connection.cc:535] client connection to 127.0.0.1:35255 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35255): Connection reset by peer (error 104)
2026-06-20T06:06:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1430 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:23 dist-test-slave-4h14 krb5kdc[25019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:23 dist-test-slave-4h14 krb5kdc[25019](info): set up 2 sockets
Jun 20 06:06:23 dist-test-slave-4h14 krb5kdc[25019](info): commencing operation
krb5kdc: starting...
W20260620 06:06:24.461146 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:24 dist-test-slave-4h14 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935584, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 06:06:24.587633 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38291
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38971
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38291
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:24.703413 25032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:24.703763 25032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:24.703866 25032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:24.708117 25032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:24.708236 25032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:24.708274 25032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:24.708335 25032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:24.708374 25032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:24.714285 25032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38971
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38291
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38291
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:24.715488 25032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:24.716595 25032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:24.725569 25037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:24.725791 25040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:24.725667 25032 server_base.cc:1034] running on GCE node
W20260620 06:06:24.725576 25038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:24.726504 25032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:24.727741 25032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:24.728916 25032 hybrid_clock.cc:648] HybridClock initialized: now 1781935584728896 us; error 34 us; skew 500 ppm
Jun 20 06:06:24 dist-test-slave-4h14 krb5kdc[25019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935584, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:24.731896 25032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 06:06:24.732947 25032 webserver.cc:460] Webserver started at http://127.0.0.1:36863/ using document root <none> and password file <none>
I20260620 06:06:24.733554 25032 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:24.733645 25032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:24.733913 25032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:24.736096 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "e831232ff50b4713a8a751c78a8605b4"
format_stamp: "Formatted at 2026-06-20 06:06:24 on dist-test-slave-4h14"
server_key: "4d214dcb0411b36a661399f3fb2bb279"
server_key_iv: "69c083bd0c732290eeb4714c9033f8bc"
server_key_version: "encryptionkey@0"
I20260620 06:06:24.736764 25032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "e831232ff50b4713a8a751c78a8605b4"
format_stamp: "Formatted at 2026-06-20 06:06:24 on dist-test-slave-4h14"
server_key: "4d214dcb0411b36a661399f3fb2bb279"
server_key_iv: "69c083bd0c732290eeb4714c9033f8bc"
server_key_version: "encryptionkey@0"
I20260620 06:06:24.740751 25032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:24.743439 25047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:24.745054 25032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:24.745224 25032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "e831232ff50b4713a8a751c78a8605b4"
format_stamp: "Formatted at 2026-06-20 06:06:24 on dist-test-slave-4h14"
server_key: "4d214dcb0411b36a661399f3fb2bb279"
server_key_iv: "69c083bd0c732290eeb4714c9033f8bc"
server_key_version: "encryptionkey@0"
I20260620 06:06:24.745362 25032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:24.750187 25032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:24.753736 25032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:24.753950 25032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:24.760851 25032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38291
I20260620 06:06:24.760872 25099 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38291 every 8 connection(s)
I20260620 06:06:24.761873 25032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:24.764633 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25032
I20260620 06:06:24.764825 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:24.765141 25100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:24.765183 3532 external_mini_cluster.cc:1428] Setting key 670b67e12e3b99404c39b3d9d1019853
I20260620 06:06:24.771395 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:24.766619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:38165 (local address 127.0.0.1:38291)
0620 06:06:24.767104 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:24.767123 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:24.767175 (+ 52us) server_negotiation.cc:407] Connection header received
0620 06:06:24.768120 (+ 945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:24.768157 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:24.768570 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:24.769009 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:24.769826 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:24.769832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:24.769848 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:24.769861 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:24.769927 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:24.769952 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:24.770098 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:24.770239 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 06:06:24.770408 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":130,"threads_started":1}
I20260620 06:06:24.773169 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4: Bootstrap starting.
I20260620 06:06:24.774755 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:24.775717 25100 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:24.776576 25100 log.cc:824] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:24.778748 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:24.776904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.120:57327 (local address 127.0.0.1:38291)
0620 06:06:24.777034 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:24.777038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:24.777336 (+ 298us) server_negotiation.cc:407] Connection header received
0620 06:06:24.777447 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:24.777450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:24.777488 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:24.777561 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:24.778304 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:24.778307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:24.778309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:24.778319 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:24.778343 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:24.778353 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:24.778423 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:24.778542 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 06:06:24.778578 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 06:06:24.779124 25100 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4: No bootstrap required, opened a new log
I20260620 06:06:24.782963 25100 raft_consensus.cc:348] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e831232ff50b4713a8a751c78a8605b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260620 06:06:24.783236 25100 raft_consensus.cc:374] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:24.783300 25100 raft_consensus.cc:729] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e831232ff50b4713a8a751c78a8605b4, State: Initialized, Role: FOLLOWER
I20260620 06:06:24.783761 25100 consensus_queue.cc:260] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e831232ff50b4713a8a751c78a8605b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260620 06:06:24.783921 25100 raft_consensus.cc:388] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:24.783998 25100 raft_consensus.cc:482] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:24.784101 25100 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:24.785454 25100 raft_consensus.cc:504] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e831232ff50b4713a8a751c78a8605b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260620 06:06:24.785838 25100 leader_election.cc:302] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e831232ff50b4713a8a751c78a8605b4; no voters:
I20260620 06:06:24.786202 25100 leader_election.cc:288] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:24.786332 25110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:24.786585 25110 raft_consensus.cc:686] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [term 1 LEADER]: Becoming Leader. State: Replica: e831232ff50b4713a8a751c78a8605b4, State: Running, Role: LEADER
I20260620 06:06:24.786975 25110 consensus_queue.cc:237] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e831232ff50b4713a8a751c78a8605b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } }
I20260620 06:06:24.787572 25100 sys_catalog.cc:564] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:24.788763 25112 sys_catalog.cc:455] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e831232ff50b4713a8a751c78a8605b4. Latest consensus state: current_term: 1 leader_uuid: "e831232ff50b4713a8a751c78a8605b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e831232ff50b4713a8a751c78a8605b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } } }
I20260620 06:06:24.788908 25112 sys_catalog.cc:458] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:24.789206 25111 sys_catalog.cc:455] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e831232ff50b4713a8a751c78a8605b4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e831232ff50b4713a8a751c78a8605b4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38291 } } }
I20260620 06:06:24.789302 25119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:24.789389 25111 sys_catalog.cc:458] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:24.792845 25119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:24.799119 25119 catalog_manager.cc:1347] Generated new cluster ID: 1f982f11b5f2435e85b8f6d957528fe2
I20260620 06:06:24.799256 25119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:24.820439 25119 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:24.820860 25119 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:24.824988 25119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e831232ff50b4713a8a751c78a8605b4: Generated new TSK 0
I20260620 06:06:24.825901 25119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:24.840394 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25032
2026-06-20T06:06:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:24 dist-test-slave-4h14 krb5kdc[25131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:24 dist-test-slave-4h14 krb5kdc[25131](info): set up 2 sockets
Jun 20 06:06:24 dist-test-slave-4h14 krb5kdc[25131](info): commencing operation
krb5kdc: starting...
W20260620 06:06:25.893038 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:25 dist-test-slave-4h14 krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:26.025137 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34277
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:34277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:26.143538 25144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:26.143925 25144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:26.144022 25144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:26.148397 25144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:26.148521 25144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:26.148558 25144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:26.148615 25144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:26.148696 25144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:26.154556 25144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:34277
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:26.155841 25144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:26.157141 25144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:26.166810 25149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:26.166807 25152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:26.166963 25150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:26.167091 25144 server_base.cc:1034] running on GCE node
I20260620 06:06:26.167841 25144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:26.169245 25144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:26.170475 25144 hybrid_clock.cc:648] HybridClock initialized: now 1781935586170439 us; error 53 us; skew 500 ppm
Jun 20 06:06:26 dist-test-slave-4h14 krb5kdc[25131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935586, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:26.173578 25144 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:26.174536 25144 webserver.cc:460] Webserver started at http://127.3.115.62:35989/ using document root <none> and password file <none>
I20260620 06:06:26.175099 25144 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:26.175161 25144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:26.175348 25144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:26.177613 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "52dd3abb6dd141f791ef601bb1611602"
format_stamp: "Formatted at 2026-06-20 06:06:26 on dist-test-slave-4h14"
server_key: "fa761066021ed92bdd04266f920db2bd"
server_key_iv: "642f67e8d069bf13ce905563beb04143"
server_key_version: "encryptionkey@0"
I20260620 06:06:26.178400 25144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "52dd3abb6dd141f791ef601bb1611602"
format_stamp: "Formatted at 2026-06-20 06:06:26 on dist-test-slave-4h14"
server_key: "fa761066021ed92bdd04266f920db2bd"
server_key_iv: "642f67e8d069bf13ce905563beb04143"
server_key_version: "encryptionkey@0"
I20260620 06:06:26.182605 25144 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 06:06:26.185438 25159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:26.186898 25144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260620 06:06:26.187014 25144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "52dd3abb6dd141f791ef601bb1611602"
format_stamp: "Formatted at 2026-06-20 06:06:26 on dist-test-slave-4h14"
server_key: "fa761066021ed92bdd04266f920db2bd"
server_key_iv: "642f67e8d069bf13ce905563beb04143"
server_key_version: "encryptionkey@0"
I20260620 06:06:26.187114 25144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:26.201428 25144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:26.205053 25144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:26.205231 25144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:26.212590 25144 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:34277
I20260620 06:06:26.212604 25211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:34277 every 8 connection(s)
I20260620 06:06:26.213706 25144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:26.216773 25212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:26.222322 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25144
I20260620 06:06:26.222481 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:26.222797 3532 external_mini_cluster.cc:1428] Setting key d05c3a4c2834f301f72e0c45b8279897
I20260620 06:06:26.222863 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602: Bootstrap starting.
I20260620 06:06:26.225600 25212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:26.226406 25212 log.cc:824] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:26.228377 25212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602: No bootstrap required, opened a new log
Jun 20 06:06:26 dist-test-slave-4h14 krb5kdc[25131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:26.232333 25212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd3abb6dd141f791ef601bb1611602" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34277 } }
I20260620 06:06:26.232607 25212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:26.232714 25212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52dd3abb6dd141f791ef601bb1611602, State: Initialized, Role: FOLLOWER
I20260620 06:06:26.233268 25212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd3abb6dd141f791ef601bb1611602" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34277 } }
I20260620 06:06:26.233474 25212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:26.233549 25212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:26.233649 25212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:26.234665 25212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd3abb6dd141f791ef601bb1611602" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34277 } }
I20260620 06:06:26.235025 25212 leader_election.cc:302] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52dd3abb6dd141f791ef601bb1611602; no voters:
I20260620 06:06:26.235363 25212 leader_election.cc:288] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:26.235491 25217 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:26.235740 25217 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [term 1 LEADER]: Becoming Leader. State: Replica: 52dd3abb6dd141f791ef601bb1611602, State: Running, Role: LEADER
I20260620 06:06:26.236110 25217 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd3abb6dd141f791ef601bb1611602" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34277 } }
I20260620 06:06:26.236819 25212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:26.237782 25218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52dd3abb6dd141f791ef601bb1611602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd3abb6dd141f791ef601bb1611602" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34277 } } }
I20260620 06:06:26.237944 25218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:26.238369 25226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:26.238369 25219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52dd3abb6dd141f791ef601bb1611602. Latest consensus state: current_term: 1 leader_uuid: "52dd3abb6dd141f791ef601bb1611602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52dd3abb6dd141f791ef601bb1611602" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34277 } } }
I20260620 06:06:26.238554 25219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:26.238713 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:26.224231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50571 (local address 127.3.115.62:34277)
0620 06:06:26.224808 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:26.224826 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:26.224865 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:06:26.225547 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:26.225564 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:26.226010 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:26.226375 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:26.227221 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:26.228231 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:26.229253 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:26.229760 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:26.229933 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:26.232725 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:26.232731 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:26.232746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:26.232776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:26.234671 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:26.235204 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:26.235210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:26.235219 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:26.235365 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:26.235834 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:26.235837 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:26.235839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:26.236343 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:26.236531 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:26.237136 (+ 605us) server_negotiation.cc:299] Negotiation successful
0620 06:06:26.237399 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":146,"threads_started":1}
I20260620 06:06:26.241767 25226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:26.242255 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:26.248721 25226 catalog_manager.cc:1347] Generated new cluster ID: 284f4d0b687f4dd1b903c2c5356e37f9
I20260620 06:06:26.248805 25226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:26.254344 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:26.244568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33237 (local address 127.3.115.62:34277)
0620 06:06:26.244710 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:26.244714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:26.245172 (+ 458us) server_negotiation.cc:407] Connection header received
0620 06:06:26.245336 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:26.245339 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:26.245393 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:26.245478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:26.246174 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:26.247282 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:26.248467 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:26.248825 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:26.248961 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:26.250465 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:26.250469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:26.250471 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:26.250499 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:26.252071 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:26.252628 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:26.252878 (+ 250us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:26.252881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:26.252985 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:26.253393 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:26.253397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:26.253399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:26.253594 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:26.253756 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:26.254044 (+ 288us) server_negotiation.cc:299] Negotiation successful
0620 06:06:26.254149 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 06:06:26.274453 25226 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:26.275812 25226 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:26.285036 25226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52dd3abb6dd141f791ef601bb1611602: Generated new TSK 0
I20260620 06:06:26.285966 25226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:26.289918 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25144
W20260620 06:06:26.295756 25129 connection.cc:535] client connection to 127.3.115.62:34277 recv error: Network error: failed to read from TLS socket (remote: 127.3.115.62:34277): Connection reset by peer (error 104)
2026-06-20T06:06:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:26 dist-test-slave-4h14 krb5kdc[25243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:26 dist-test-slave-4h14 krb5kdc[25243](info): set up 2 sockets
Jun 20 06:06:26 dist-test-slave-4h14 krb5kdc[25243](info): commencing operation
krb5kdc: starting...
W20260620 06:06:27.333143 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:27 dist-test-slave-4h14 krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935587, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:27.463420 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:38395
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:33215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:38395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:27.578190 25256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:27.578552 25256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:27.578636 25256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:27.582911 25256 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:27.583021 25256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:27.583062 25256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:27.583108 25256 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:27.583144 25256 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:27.588936 25256 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:38395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:38395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:27.590137 25256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:27.591235 25256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:27.600199 25262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:27.600199 25264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:27.600193 25261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:27.600643 25256 server_base.cc:1034] running on GCE node
I20260620 06:06:27.601303 25256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:27.602509 25256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:27.603734 25256 hybrid_clock.cc:648] HybridClock initialized: now 1781935587603710 us; error 40 us; skew 500 ppm
Jun 20 06:06:27 dist-test-slave-4h14 krb5kdc[25243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935587, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:27.606822 25256 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:27.607795 25256 webserver.cc:460] Webserver started at http://127.3.115.62:43855/ using document root <none> and password file <none>
I20260620 06:06:27.608388 25256 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:27.608484 25256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:27.608771 25256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:27.610903 25256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "964a2be4d4fc44ee85bcc479f50d9a4a"
format_stamp: "Formatted at 2026-06-20 06:06:27 on dist-test-slave-4h14"
server_key: "6c5d6203395d5f2bfb8138fe94058a0b"
server_key_iv: "659187a95ec85465a7cfe5caa977ee5c"
server_key_version: "encryptionkey@0"
I20260620 06:06:27.611497 25256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "964a2be4d4fc44ee85bcc479f50d9a4a"
format_stamp: "Formatted at 2026-06-20 06:06:27 on dist-test-slave-4h14"
server_key: "6c5d6203395d5f2bfb8138fe94058a0b"
server_key_iv: "659187a95ec85465a7cfe5caa977ee5c"
server_key_version: "encryptionkey@0"
I20260620 06:06:27.615419 25256 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:27.618245 25271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:27.619740 25256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:27.619907 25256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "964a2be4d4fc44ee85bcc479f50d9a4a"
format_stamp: "Formatted at 2026-06-20 06:06:27 on dist-test-slave-4h14"
server_key: "6c5d6203395d5f2bfb8138fe94058a0b"
server_key_iv: "659187a95ec85465a7cfe5caa977ee5c"
server_key_version: "encryptionkey@0"
I20260620 06:06:27.620044 25256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:27.643414 25256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:27.647333 25256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:27.647586 25256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:27.655320 25256 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:38395
I20260620 06:06:27.655342 25323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:38395 every 8 connection(s)
I20260620 06:06:27.656378 25256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:27.659353 25324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:27.660640 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25256
I20260620 06:06:27.660830 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:27.661162 3532 external_mini_cluster.cc:1428] Setting key 4677482913777501d1ab12d4be2fa021
I20260620 06:06:27.666196 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a: Bootstrap starting.
I20260620 06:06:27.668655 25324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:27.669548 25324 log.cc:824] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:27.671556 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:27.662828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58965 (local address 127.3.115.62:38395)
0620 06:06:27.663345 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:27.663365 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:27.663405 (+ 40us) server_negotiation.cc:407] Connection header received
0620 06:06:27.664141 (+ 736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:27.664162 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:27.664593 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:27.665083 (+ 490us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:27.665540 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:27.666634 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:27.667592 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:27.668113 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:27.668299 (+ 186us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:27.669361 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:27.669366 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:27.669381 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:27.669903 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:27.669991 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:27.670027 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:27.670185 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:27.670405 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 06:06:27.670560 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":145,"threads_started":1}
I20260620 06:06:27.672005 25324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a: No bootstrap required, opened a new log
I20260620 06:06:27.675187 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:27.675752 25324 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 38395 } }
I20260620 06:06:27.676000 25324 raft_consensus.cc:374] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:27.676060 25324 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964a2be4d4fc44ee85bcc479f50d9a4a, State: Initialized, Role: FOLLOWER
I20260620 06:06:27.676625 25324 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 38395 } }
I20260620 06:06:27.676829 25324 raft_consensus.cc:388] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:27.676915 25324 raft_consensus.cc:482] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:27.677016 25324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:27.678234 25324 raft_consensus.cc:504] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 38395 } }
I20260620 06:06:27.678637 25324 leader_election.cc:302] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 964a2be4d4fc44ee85bcc479f50d9a4a; no voters:
I20260620 06:06:27.679011 25324 leader_election.cc:288] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:27.679083 25334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:27.679311 25334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [term 1 LEADER]: Becoming Leader. State: Replica: 964a2be4d4fc44ee85bcc479f50d9a4a, State: Running, Role: LEADER
I20260620 06:06:27.679716 25334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 38395 } }
I20260620 06:06:27.680496 25324 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:27.681644 25336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 38395 } } }
I20260620 06:06:27.681844 25336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:27.682423 25327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:27.677595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53867 (local address 127.3.115.62:38395)
0620 06:06:27.677791 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:27.677800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:27.677869 (+ 69us) server_negotiation.cc:407] Connection header received
0620 06:06:27.678083 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:27.678087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:27.678135 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:27.678236 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:27.678580 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:27.679312 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:27.680314 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:27.680626 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:27.680750 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:27.681768 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:27.681772 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:27.681775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:27.681789 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:27.681820 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:27.681837 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:27.681906 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:27.682167 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 06:06:27.682209 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260620 06:06:27.682214 25335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964a2be4d4fc44ee85bcc479f50d9a4a. Latest consensus state: current_term: 1 leader_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964a2be4d4fc44ee85bcc479f50d9a4a" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 38395 } } }
I20260620 06:06:27.683315 25335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a [sys.catalog]: This master's current role is: LEADER
W20260620 06:06:27.686082 25349 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:27.686162 25349 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:27.686245 25343 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:27.687072 25343 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:27.692562 25343 catalog_manager.cc:1347] Generated new cluster ID: ef7db213633b44f1ab273f53839e5184
I20260620 06:06:27.692649 25343 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:27.707649 25343 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:27.708776 25343 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:27.716315 25343 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 964a2be4d4fc44ee85bcc479f50d9a4a: Generated new TSK 0
I20260620 06:06:27.717283 25343 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:27.731158 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25256
W20260620 06:06:27.736032 25241 connection.cc:535] client connection to 127.3.115.62:38395 recv error: Network error: failed to read from TLS socket (remote: 127.3.115.62:38395): Connection reset by peer (error 104)
2026-06-20T06:06:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:27 dist-test-slave-4h14 krb5kdc[25355](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:27 dist-test-slave-4h14 krb5kdc[25355](info): set up 2 sockets
Jun 20 06:06:27 dist-test-slave-4h14 krb5kdc[25355](info): commencing operation
krb5kdc: starting...
W20260620 06:06:28.769197 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:28 dist-test-slave-4h14 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:28.896956 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42091
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:36641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:29.011302 25368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:29.011669 25368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:29.011759 25368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:29.015905 25368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:29.016019 25368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:29.016052 25368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:29.016081 25368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:29.016116 25368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:29.021821 25368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:36641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:42091
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25368
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:29.023015 25368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:29.024097 25368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:29.033080 25374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:29.033221 25368 server_base.cc:1034] running on GCE node
W20260620 06:06:29.033120 25373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:29.033088 25376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:29.034029 25368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:29.035157 25368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:29.036337 25368 hybrid_clock.cc:648] HybridClock initialized: now 1781935589036316 us; error 30 us; skew 500 ppm
Jun 20 06:06:29 dist-test-slave-4h14 krb5kdc[25355](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935589, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:29.044903 25368 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:29.045984 25368 webserver.cc:460] Webserver started at http://127.3.115.62:40371/ using document root <none> and password file <none>
I20260620 06:06:29.046608 25368 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:29.046716 25368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:29.046960 25368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:29.049248 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "8b2359a3261743eea8065eb34e1e646e"
format_stamp: "Formatted at 2026-06-20 06:06:29 on dist-test-slave-4h14"
server_key: "78e9edb442d0489623a24561d2c1ad20"
server_key_iv: "09395f68c6b886319fb380ec20484028"
server_key_version: "encryptionkey@0"
I20260620 06:06:29.049866 25368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "8b2359a3261743eea8065eb34e1e646e"
format_stamp: "Formatted at 2026-06-20 06:06:29 on dist-test-slave-4h14"
server_key: "78e9edb442d0489623a24561d2c1ad20"
server_key_iv: "09395f68c6b886319fb380ec20484028"
server_key_version: "encryptionkey@0"
I20260620 06:06:29.053834 25368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260620 06:06:29.056437 25383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:29.057943 25368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:29.058108 25368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "8b2359a3261743eea8065eb34e1e646e"
format_stamp: "Formatted at 2026-06-20 06:06:29 on dist-test-slave-4h14"
server_key: "78e9edb442d0489623a24561d2c1ad20"
server_key_iv: "09395f68c6b886319fb380ec20484028"
server_key_version: "encryptionkey@0"
I20260620 06:06:29.058250 25368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:29.063153 25368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:29.066664 25368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:29.066865 25368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:29.074240 25368 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:42091
I20260620 06:06:29.074307 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:42091 every 8 connection(s)
I20260620 06:06:29.075219 25368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:29.078091 25436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:29.083770 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25368
I20260620 06:06:29.083777 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e: Bootstrap starting.
I20260620 06:06:29.083937 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:29.084233 3532 external_mini_cluster.cc:1428] Setting key 52c3c79e68fa62bc09886f4bf8eb870a
I20260620 06:06:29.086529 25436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:29.087288 25436 log.cc:824] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:29.089576 25436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e: No bootstrap required, opened a new log
I20260620 06:06:29.090538 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:29.085621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49269 (local address 127.3.115.62:42091)
0620 06:06:29.086042 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:29.086057 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:29.086095 (+ 38us) server_negotiation.cc:407] Connection header received
0620 06:06:29.086873 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:29.086893 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:29.087224 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:29.087614 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:29.089039 (+ 1425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:29.089045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:29.089061 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:29.089073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:29.089142 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:29.089169 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:29.089286 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:29.089442 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 06:06:29.089609 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":141,"threads_started":1}
I20260620 06:06:29.093698 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:29.093990 25436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2359a3261743eea8065eb34e1e646e" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42091 } }
I20260620 06:06:29.094230 25436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:29.094302 25436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b2359a3261743eea8065eb34e1e646e, State: Initialized, Role: FOLLOWER
I20260620 06:06:29.094734 25436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2359a3261743eea8065eb34e1e646e" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42091 } }
I20260620 06:06:29.094949 25436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:29.095091 25436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:29.095224 25436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:29.096323 25436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2359a3261743eea8065eb34e1e646e" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42091 } }
I20260620 06:06:29.096765 25436 leader_election.cc:302] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b2359a3261743eea8065eb34e1e646e; no voters:
I20260620 06:06:29.097123 25436 leader_election.cc:288] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:29.097440 25446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:29.097712 25446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [term 1 LEADER]: Becoming Leader. State: Replica: 8b2359a3261743eea8065eb34e1e646e, State: Running, Role: LEADER
I20260620 06:06:29.098073 25439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:29.096004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36455 (local address 127.3.115.62:42091)
0620 06:06:29.096150 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:29.096155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:29.096375 (+ 220us) server_negotiation.cc:407] Connection header received
0620 06:06:29.096531 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:29.096534 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:29.096587 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:29.096702 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:29.097602 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:29.097607 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:29.097609 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:29.097624 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:29.097655 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:29.097673 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:29.097747 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:29.097917 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 06:06:29.097959 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:29.098300 25446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2359a3261743eea8065eb34e1e646e" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42091 } }
I20260620 06:06:29.098587 25436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:29.100075 25447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b2359a3261743eea8065eb34e1e646e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2359a3261743eea8065eb34e1e646e" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42091 } } }
I20260620 06:06:29.100211 25447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:29.100565 25448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b2359a3261743eea8065eb34e1e646e. Latest consensus state: current_term: 1 leader_uuid: "8b2359a3261743eea8065eb34e1e646e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b2359a3261743eea8065eb34e1e646e" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 42091 } } }
I20260620 06:06:29.100620 25457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:29.100687 25448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:29.103506 25457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:29.108927 25457 catalog_manager.cc:1347] Generated new cluster ID: 87e133b2ce7543fdab4273f66b8b37a7
I20260620 06:06:29.109004 25457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:29.138967 25457 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:29.139430 25457 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:29.151729 25457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b2359a3261743eea8065eb34e1e646e: Generated new TSK 0
I20260620 06:06:29.152698 25457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:29.162784 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25368
2026-06-20T06:06:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1432 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:29 dist-test-slave-4h14 krb5kdc[25467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:29 dist-test-slave-4h14 krb5kdc[25467](info): set up 2 sockets
Jun 20 06:06:29 dist-test-slave-4h14 krb5kdc[25467](info): commencing operation
krb5kdc: starting...
W20260620 06:06:30.217119 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:30 dist-test-slave-4h14 krb5kdc[25467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:30.347846 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43875
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:34531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:30.462497 25480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:30.462810 25480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:30.462855 25480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:30.467037 25480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:30.467119 25480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:30.467139 25480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:30.467157 25480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:30.467173 25480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:30.472774 25480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:34531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:30.473965 25480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:30.475059 25480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:30.484100 25488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:30.484472 25480 server_base.cc:1034] running on GCE node
W20260620 06:06:30.484130 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:30.484100 25485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:30.485392 25480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:30.486670 25480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:30.487866 25480 hybrid_clock.cc:648] HybridClock initialized: now 1781935590487852 us; error 30 us; skew 500 ppm
Jun 20 06:06:30 dist-test-slave-4h14 krb5kdc[25467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935590, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:30.490782 25480 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:30.491822 25480 webserver.cc:460] Webserver started at http://127.3.115.62:35297/ using document root <none> and password file <none>
I20260620 06:06:30.492422 25480 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:30.492524 25480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:30.492822 25480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:30.495002 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "539bce7c971144b7a366fcfcc46c5c75"
format_stamp: "Formatted at 2026-06-20 06:06:30 on dist-test-slave-4h14"
server_key: "7062d9794d1ddec742ee7a8e44e810da"
server_key_iv: "70f63b6e28064b64ee5fe71b2ac9ada2"
server_key_version: "encryptionkey@0"
I20260620 06:06:30.495616 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "539bce7c971144b7a366fcfcc46c5c75"
format_stamp: "Formatted at 2026-06-20 06:06:30 on dist-test-slave-4h14"
server_key: "7062d9794d1ddec742ee7a8e44e810da"
server_key_iv: "70f63b6e28064b64ee5fe71b2ac9ada2"
server_key_version: "encryptionkey@0"
I20260620 06:06:30.499812 25480 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260620 06:06:30.502526 25495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:30.503954 25480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:30.504114 25480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "539bce7c971144b7a366fcfcc46c5c75"
format_stamp: "Formatted at 2026-06-20 06:06:30 on dist-test-slave-4h14"
server_key: "7062d9794d1ddec742ee7a8e44e810da"
server_key_iv: "70f63b6e28064b64ee5fe71b2ac9ada2"
server_key_version: "encryptionkey@0"
I20260620 06:06:30.504251 25480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:30.520083 25480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:30.524051 25480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:30.524312 25480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:30.532163 25480 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:43875
I20260620 06:06:30.532182 25547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:43875 every 8 connection(s)
I20260620 06:06:30.533386 25480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:30.534917 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25480
I20260620 06:06:30.535068 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:30.535395 3532 external_mini_cluster.cc:1428] Setting key 5a48f3536737f4ed68c450a46ec23af0
I20260620 06:06:30.536820 25548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:30.542754 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75: Bootstrap starting.
Jun 20 06:06:30 dist-test-slave-4h14 krb5kdc[25467](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935590, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:30.545652 25548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:30.546752 25548 log.cc:824] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:30.549319 25548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75: No bootstrap required, opened a new log
I20260620 06:06:30.550652 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:30.536918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38027 (local address 127.3.115.62:43875)
0620 06:06:30.537366 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:30.537382 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:30.537421 (+ 39us) server_negotiation.cc:407] Connection header received
0620 06:06:30.538098 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:30.538116 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:30.538463 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:30.538818 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:30.539649 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:30.540757 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:30.541633 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:30.542128 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:30.542304 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:30.545285 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:30.545292 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:30.545310 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:30.545346 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:30.547482 (+ 2136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:30.547954 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:30.547961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:30.547967 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:30.548124 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:30.548389 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:30.548395 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:30.548400 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:30.548938 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:30.549149 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:30.549357 (+ 208us) server_negotiation.cc:299] Negotiation successful
0620 06:06:30.549673 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":147,"threads_started":1}
I20260620 06:06:30.553391 25548 raft_consensus.cc:348] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539bce7c971144b7a366fcfcc46c5c75" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43875 } }
I20260620 06:06:30.553624 25548 raft_consensus.cc:374] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:30.553707 25548 raft_consensus.cc:729] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 539bce7c971144b7a366fcfcc46c5c75, State: Initialized, Role: FOLLOWER
I20260620 06:06:30.554196 25548 consensus_queue.cc:260] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539bce7c971144b7a366fcfcc46c5c75" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43875 } }
I20260620 06:06:30.554282 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:30.554334 25548 raft_consensus.cc:388] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:30.554414 25548 raft_consensus.cc:482] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:30.554538 25548 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:30.555486 25548 raft_consensus.cc:504] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539bce7c971144b7a366fcfcc46c5c75" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43875 } }
I20260620 06:06:30.555874 25548 leader_election.cc:302] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 539bce7c971144b7a366fcfcc46c5c75; no voters:
I20260620 06:06:30.556180 25548 leader_election.cc:288] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:30.556305 25557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:30.556548 25557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [term 1 LEADER]: Becoming Leader. State: Replica: 539bce7c971144b7a366fcfcc46c5c75, State: Running, Role: LEADER
I20260620 06:06:30.556914 25557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539bce7c971144b7a366fcfcc46c5c75" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43875 } }
I20260620 06:06:30.557483 25548 sys_catalog.cc:564] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:30.558046 25558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 539bce7c971144b7a366fcfcc46c5c75. Latest consensus state: current_term: 1 leader_uuid: "539bce7c971144b7a366fcfcc46c5c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539bce7c971144b7a366fcfcc46c5c75" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43875 } } }
I20260620 06:06:30.558243 25558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:30.558984 25559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "539bce7c971144b7a366fcfcc46c5c75" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "539bce7c971144b7a366fcfcc46c5c75" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43875 } } }
I20260620 06:06:30.559158 25559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:30.560139 25566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:30.563966 25566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:30.567478 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:30.556545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60163 (local address 127.3.115.62:43875)
0620 06:06:30.556710 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:30.556714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:30.558214 (+ 1500us) server_negotiation.cc:407] Connection header received
0620 06:06:30.558413 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:30.558417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:30.558464 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:30.558521 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:30.559558 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:30.560500 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:30.561419 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:30.561822 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:30.561934 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:30.563708 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:30.563712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:30.563715 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:30.563743 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:30.565576 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:30.566092 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:30.566097 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:30.566099 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:30.566192 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:30.566527 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:30.566530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:30.566532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:30.566764 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:30.566895 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:30.567191 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 06:06:30.567323 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 06:06:30.570964 25566 catalog_manager.cc:1347] Generated new cluster ID: 7cd4fdf4c36d477b9399d77d26441b25
I20260620 06:06:30.571072 25566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:30.587203 25566 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:30.588253 25566 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:30.600215 25566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 539bce7c971144b7a366fcfcc46c5c75: Generated new TSK 0
I20260620 06:06:30.601222 25566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:30.611882 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25480
2026-06-20T06:06:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:30 dist-test-slave-4h14 krb5kdc[25579](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:30 dist-test-slave-4h14 krb5kdc[25579](info): set up 2 sockets
Jun 20 06:06:30 dist-test-slave-4h14 krb5kdc[25579](info): commencing operation
krb5kdc: starting...
W20260620 06:06:31.653276 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:31 dist-test-slave-4h14 krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:31.779994 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34793
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:34793
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:31.896281 25592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:31.896592 25592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:31.896637 25592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:31.900878 25592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:31.901002 25592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:31.901044 25592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:31.901082 25592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:31.901119 25592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:31.906792 25592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:34793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:34793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:31.908039 25592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:31.909227 25592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:31.918576 25598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:31.918678 25597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:31.918568 25600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:31.918872 25592 server_base.cc:1034] running on GCE node
I20260620 06:06:31.919667 25592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:31.921103 25592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:31.922333 25592 hybrid_clock.cc:648] HybridClock initialized: now 1781935591922291 us; error 57 us; skew 500 ppm
Jun 20 06:06:31 dist-test-slave-4h14 krb5kdc[25579](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935591, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:31.925417 25592 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:31.926486 25592 webserver.cc:460] Webserver started at http://127.3.115.62:46343/ using document root <none> and password file <none>
I20260620 06:06:31.927114 25592 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:31.927184 25592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:31.927413 25592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:31.929816 25592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "b4eb7feafb934f8890aaf004876dd582"
format_stamp: "Formatted at 2026-06-20 06:06:31 on dist-test-slave-4h14"
server_key: "3d4b660880261d32433c48484c85a967"
server_key_iv: "011b65583a9e8a25659ce7153602bf92"
server_key_version: "encryptionkey@0"
I20260620 06:06:31.930439 25592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "b4eb7feafb934f8890aaf004876dd582"
format_stamp: "Formatted at 2026-06-20 06:06:31 on dist-test-slave-4h14"
server_key: "3d4b660880261d32433c48484c85a967"
server_key_iv: "011b65583a9e8a25659ce7153602bf92"
server_key_version: "encryptionkey@0"
I20260620 06:06:31.934753 25592 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260620 06:06:31.937569 25607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:31.938977 25592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260620 06:06:31.939127 25592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "b4eb7feafb934f8890aaf004876dd582"
format_stamp: "Formatted at 2026-06-20 06:06:31 on dist-test-slave-4h14"
server_key: "3d4b660880261d32433c48484c85a967"
server_key_iv: "011b65583a9e8a25659ce7153602bf92"
server_key_version: "encryptionkey@0"
I20260620 06:06:31.939260 25592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:31.950090 25592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:31.953332 25592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:31.953554 25592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:31.960856 25592 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:34793
I20260620 06:06:31.960886 25659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:34793 every 8 connection(s)
I20260620 06:06:31.961905 25592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:31.965036 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:31.966913 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25592
I20260620 06:06:31.967031 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:31.967317 3532 external_mini_cluster.cc:1428] Setting key 17614c22aa0c37186916626266af834d
I20260620 06:06:31.972505 25660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582: Bootstrap starting.
I20260620 06:06:31.975456 25660 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:31.976421 25660 log.cc:824] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:31.978022 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:31.968931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34767 (local address 127.3.115.62:34793)
0620 06:06:31.969421 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:31.969441 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:31.969489 (+ 48us) server_negotiation.cc:407] Connection header received
0620 06:06:31.970430 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:31.970468 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:31.970969 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:31.971400 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:31.971848 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:31.973308 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:31.974262 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:31.974973 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:31.975143 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:31.976033 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:31.976039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:31.976054 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:31.976483 (+ 429us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:31.976545 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:31.976604 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:31.976774 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:31.976956 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 06:06:31.977215 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":141,"threads_started":1}
I20260620 06:06:31.978888 25660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582: No bootstrap required, opened a new log
I20260620 06:06:31.981166 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:31.983217 25660 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb7feafb934f8890aaf004876dd582" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34793 } }
I20260620 06:06:31.983549 25660 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:31.983619 25660 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4eb7feafb934f8890aaf004876dd582, State: Initialized, Role: FOLLOWER
I20260620 06:06:31.984134 25660 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb7feafb934f8890aaf004876dd582" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34793 } }
I20260620 06:06:31.984267 25660 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:31.984354 25660 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:31.984491 25660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:31.985532 25660 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb7feafb934f8890aaf004876dd582" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34793 } }
I20260620 06:06:31.985909 25660 leader_election.cc:302] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4eb7feafb934f8890aaf004876dd582; no voters:
I20260620 06:06:31.986299 25660 leader_election.cc:288] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:31.986372 25670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:31.986627 25670 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [term 1 LEADER]: Becoming Leader. State: Replica: b4eb7feafb934f8890aaf004876dd582, State: Running, Role: LEADER
I20260620 06:06:31.987010 25670 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb7feafb934f8890aaf004876dd582" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34793 } }
I20260620 06:06:31.987584 25663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:31.983305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50177 (local address 127.3.115.62:34793)
0620 06:06:31.983532 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:31.983538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:31.983553 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:31.983740 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:31.983744 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:31.983800 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:31.983886 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:31.984152 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:31.984940 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:31.985790 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:31.986089 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:31.986201 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:31.987112 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:31.987119 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:31.987122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:31.987138 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:31.987172 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:31.987186 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:31.987248 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:31.987387 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 06:06:31.987444 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260620 06:06:31.987800 25660 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:31.989151 25671 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4eb7feafb934f8890aaf004876dd582. Latest consensus state: current_term: 1 leader_uuid: "b4eb7feafb934f8890aaf004876dd582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb7feafb934f8890aaf004876dd582" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34793 } } }
I20260620 06:06:31.989369 25671 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:31.989573 25672 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4eb7feafb934f8890aaf004876dd582" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4eb7feafb934f8890aaf004876dd582" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 34793 } } }
I20260620 06:06:31.989929 25672 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:31.990079 25684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:31.994170 25684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:32.000944 25684 catalog_manager.cc:1347] Generated new cluster ID: 55f1c7f9bfe44bea9104e93b195d90f0
I20260620 06:06:32.001045 25684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:32.006870 25684 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:32.008008 25684 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:32.012451 25684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4eb7feafb934f8890aaf004876dd582: Generated new TSK 0
I20260620 06:06:32.013402 25684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:32.026319 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25592
2026-06-20T06:06:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:32 dist-test-slave-4h14 krb5kdc[25691](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:32 dist-test-slave-4h14 krb5kdc[25691](info): set up 2 sockets
Jun 20 06:06:32 dist-test-slave-4h14 krb5kdc[25691](info): commencing operation
krb5kdc: starting...
W20260620 06:06:33.069123 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:33 dist-test-slave-4h14 krb5kdc[25691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:33.193845 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:41299
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:41585
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:41299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:33.315461 25704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:33.315910 25704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:33.316001 25704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:33.321729 25704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:33.321815 25704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:33.321837 25704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:33.321856 25704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:33.321871 25704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:33.327725 25704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:41585
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:41299
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:41299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:33.329027 25704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:33.330132 25704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:33.339542 25710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:33.339692 25704 server_base.cc:1034] running on GCE node
W20260620 06:06:33.339550 25712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:33.339600 25709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:33.340586 25704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:33.341899 25704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:33.343083 25704 hybrid_clock.cc:648] HybridClock initialized: now 1781935593343069 us; error 29 us; skew 500 ppm
Jun 20 06:06:33 dist-test-slave-4h14 krb5kdc[25691](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935593, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:33.346263 25704 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:33.347347 25704 webserver.cc:460] Webserver started at http://127.3.115.62:41145/ using document root <none> and password file <none>
I20260620 06:06:33.347936 25704 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:33.348007 25704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:33.348189 25704 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:33.350553 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "c76d42a5baff49a8bd263ba87038cda6"
format_stamp: "Formatted at 2026-06-20 06:06:33 on dist-test-slave-4h14"
server_key: "e793bfd01e66120a6b64fd813e19fa33"
server_key_iv: "3841a08f1ab745fb007202aad71a37b1"
server_key_version: "encryptionkey@0"
I20260620 06:06:33.351161 25704 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "c76d42a5baff49a8bd263ba87038cda6"
format_stamp: "Formatted at 2026-06-20 06:06:33 on dist-test-slave-4h14"
server_key: "e793bfd01e66120a6b64fd813e19fa33"
server_key_iv: "3841a08f1ab745fb007202aad71a37b1"
server_key_version: "encryptionkey@0"
I20260620 06:06:33.355657 25704 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:33.358458 25719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:33.359968 25704 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:33.360141 25704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "c76d42a5baff49a8bd263ba87038cda6"
format_stamp: "Formatted at 2026-06-20 06:06:33 on dist-test-slave-4h14"
server_key: "e793bfd01e66120a6b64fd813e19fa33"
server_key_iv: "3841a08f1ab745fb007202aad71a37b1"
server_key_version: "encryptionkey@0"
I20260620 06:06:33.360271 25704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:33.365437 25704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:33.368609 25704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:33.368868 25704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:33.376232 25704 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:41299
I20260620 06:06:33.376250 25771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:41299 every 8 connection(s)
I20260620 06:06:33.377336 25704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:33.380252 25772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:33.380962 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25704
I20260620 06:06:33.381103 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:33.381385 3532 external_mini_cluster.cc:1428] Setting key cdb995fa344c3820414ed7ab1433d019
I20260620 06:06:33.387710 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:33.382943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44255 (local address 127.3.115.62:41299)
0620 06:06:33.383483 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:33.383504 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:33.383555 (+ 51us) server_negotiation.cc:407] Connection header received
0620 06:06:33.384433 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:33.384460 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:33.384895 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:33.385313 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:33.386172 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:33.386178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:33.386195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:33.386207 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:33.386275 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:33.386303 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:33.386463 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:33.386605 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 06:06:33.386773 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":148,"threads_started":1}
I20260620 06:06:33.387729 25772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6: Bootstrap starting.
I20260620 06:06:33.390300 25772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:33.391109 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:33.391259 25772 log.cc:824] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:33.393800 25772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6: No bootstrap required, opened a new log
I20260620 06:06:33.395525 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:33.393837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46245 (local address 127.3.115.62:41299)
0620 06:06:33.393978 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:33.393984 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:33.393997 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:33.394043 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:33.394045 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:33.394075 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:33.394168 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:33.395028 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:33.395033 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:33.395035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 06:06:33.395050 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:33.395082 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 06:06:33.395092 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:33.395191 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:33.395288 (+ 97us) server_negotiation.cc:299] Negotiation successful
0620 06:06:33.395333 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:06:33.398358 25772 raft_consensus.cc:348] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76d42a5baff49a8bd263ba87038cda6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41299 } }
I20260620 06:06:33.398691 25772 raft_consensus.cc:374] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 06:06:33.398763 25772 raft_consensus.cc:729] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76d42a5baff49a8bd263ba87038cda6, State: Initialized, Role: FOLLOWER
I20260620 06:06:33.399295 25772 consensus_queue.cc:260] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76d42a5baff49a8bd263ba87038cda6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41299 } }
I20260620 06:06:33.399459 25772 raft_consensus.cc:388] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 06:06:33.399551 25772 raft_consensus.cc:482] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 06:06:33.399668 25772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:33.400831 25772 raft_consensus.cc:504] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76d42a5baff49a8bd263ba87038cda6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41299 } }
I20260620 06:06:33.401266 25772 leader_election.cc:302] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c76d42a5baff49a8bd263ba87038cda6; no voters:
I20260620 06:06:33.401726 25772 leader_election.cc:288] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 06:06:33.401889 25782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:33.402133 25782 raft_consensus.cc:686] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [term 1 LEADER]: Becoming Leader. State: Replica: c76d42a5baff49a8bd263ba87038cda6, State: Running, Role: LEADER
I20260620 06:06:33.402478 25782 consensus_queue.cc:237] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76d42a5baff49a8bd263ba87038cda6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41299 } }
I20260620 06:06:33.403101 25772 sys_catalog.cc:564] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:33.404271 25783 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c76d42a5baff49a8bd263ba87038cda6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76d42a5baff49a8bd263ba87038cda6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41299 } } }
I20260620 06:06:33.404436 25783 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:33.404788 25784 sys_catalog.cc:455] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c76d42a5baff49a8bd263ba87038cda6. Latest consensus state: current_term: 1 leader_uuid: "c76d42a5baff49a8bd263ba87038cda6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c76d42a5baff49a8bd263ba87038cda6" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 41299 } } }
I20260620 06:06:33.404891 25784 sys_catalog.cc:458] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:33.404881 25791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:33.408316 25791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:33.415155 25791 catalog_manager.cc:1347] Generated new cluster ID: f74d0e922cbf47adb96e5cf57af80f3d
I20260620 06:06:33.415297 25791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:33.436619 25791 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:33.437103 25791 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:33.447165 25791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c76d42a5baff49a8bd263ba87038cda6: Generated new TSK 0
I20260620 06:06:33.448163 25791 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 06:06:33.453015 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25704
2026-06-20T06:06:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1426 ms)
[----------] 18 tests from AuthTokenIssuingTest (25823 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:33 dist-test-slave-4h14 krb5kdc[25803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:33 dist-test-slave-4h14 krb5kdc[25803](info): set up 2 sockets
Jun 20 06:06:33 dist-test-slave-4h14 krb5kdc[25803](info): commencing operation
krb5kdc: starting...
W20260620 06:06:34.501143 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:34 dist-test-slave-4h14 krb5kdc[25803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:34.631656 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37159
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:37159,127.3.115.61:45443,127.3.115.60:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:34.749315 25816 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:06:34.749707 25816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:34.749784 25816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:34.749869 25816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:34.754096 25816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:34.754217 25816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:34.754256 25816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:34.754297 25816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:34.754333 25816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:34.759958 25816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:37159,127.3.115.61:45443,127.3.115.60:37599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:37159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:34.761258 25816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:34.762395 25816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:34.771919 25822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:34.771952 25821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:34.772061 25816 server_base.cc:1034] running on GCE node
W20260620 06:06:34.771919 25824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:34.772987 25816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:34.774228 25816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:34.775408 25816 hybrid_clock.cc:648] HybridClock initialized: now 1781935594775396 us; error 31 us; skew 500 ppm
Jun 20 06:06:34 dist-test-slave-4h14 krb5kdc[25803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:34.783077 25816 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:34.784188 25816 webserver.cc:460] Webserver started at http://127.3.115.62:44009/ using document root <none> and password file <none>
I20260620 06:06:34.784909 25816 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:34.784983 25816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:34.785168 25816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:34.787361 25816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "d2446777aabc44658e70c7f91d7e7e07"
format_stamp: "Formatted at 2026-06-20 06:06:34 on dist-test-slave-4h14"
server_key: "f0e7b8dbd6a201ececa1a9db512c3dfb"
server_key_iv: "a993a6a2855edf4d15f2dde2befb121d"
server_key_version: "encryptionkey@0"
I20260620 06:06:34.787959 25816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "d2446777aabc44658e70c7f91d7e7e07"
format_stamp: "Formatted at 2026-06-20 06:06:34 on dist-test-slave-4h14"
server_key: "f0e7b8dbd6a201ececa1a9db512c3dfb"
server_key_iv: "a993a6a2855edf4d15f2dde2befb121d"
server_key_version: "encryptionkey@0"
I20260620 06:06:34.792132 25816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:34.794951 25831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:34.796358 25816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260620 06:06:34.796572 25816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "d2446777aabc44658e70c7f91d7e7e07"
format_stamp: "Formatted at 2026-06-20 06:06:34 on dist-test-slave-4h14"
server_key: "f0e7b8dbd6a201ececa1a9db512c3dfb"
server_key_iv: "a993a6a2855edf4d15f2dde2befb121d"
server_key_version: "encryptionkey@0"
I20260620 06:06:34.796774 25816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:34.814715 25816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:34.818374 25816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:34.818651 25816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:34.826200 25816 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:37159
I20260620 06:06:34.826712 25883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:37159 every 8 connection(s)
I20260620 06:06:34.827292 25816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:34.828909 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25816
I20260620 06:06:34.829066 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:34.829350 3532 external_mini_cluster.cc:1428] Setting key dacd92f1fc882bc6c68b83f17b0617d1
I20260620 06:06:34.830809 25884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:34.835186 25884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.61@KRBTEST.COM" created.
Jun 20 06:06:34 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:34.853070 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.838928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45122 (local address 127.3.115.62:37159)
0620 06:06:34.839575 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:34.839589 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:34.839641 (+ 52us) server_negotiation.cc:407] Connection header received
0620 06:06:34.840420 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:34.840429 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:34.840731 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:34.840918 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:34.842310 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:34.843298 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:34.844416 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:34.845157 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:34.845371 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:34.847826 (+ 2455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:34.847836 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:34.847856 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:34.847891 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:34.849714 (+ 1823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:34.850269 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:34.850275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:34.850276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:34.850388 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:34.850741 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:34.850746 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:34.850751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:34.851287 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:34.851431 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:34.851849 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 06:06:34.852061 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":97,"threads_started":1}
I20260620 06:06:34.853089 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.837911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37159 (local address 127.0.0.1:45122)
0620 06:06:34.838559 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.838648 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:34.839977 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:34.841449 (+ 1472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:34.841469 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:34.841891 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:34.842150 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:34.842165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:34.843428 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:34.843431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:34.844240 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:34.844254 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:34.845370 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:34.845374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:34.845488 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:34.846155 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:34.846173 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:34.847648 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:34.849822 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:34.849831 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:34.849840 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:34.850161 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:34.850480 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:34.850483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:34.850484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:34.850634 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:34.851410 (+ 776us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:34.851416 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:34.851604 (+ 188us) client_negotiation.cc:769] Sending connection context
0620 06:06:34.851759 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 06:06:34.852164 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":91,"spinlock_wait_cycles":768,"thread_start_us":145,"threads_started":1}
I20260620 06:06:34.855772 25884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab.
Entry for principal kudu/127.3.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab.
I20260620 06:06:34.856611 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.856303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53304)
0620 06:06:34.856466 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.856541 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260620 06:06:34.856870 25835 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
I20260620 06:06:34.857517 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:45443
--webserver_interface=127.3.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab
--principal=kudu/127.3.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:37159,127.3.115.61:45443,127.3.115.60:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:06:34.857815 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.857568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53306)
0620 06:06:34.857708 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.857761 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260620 06:06:34.858001 25884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:45443: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
I20260620 06:06:34.907315 25884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } attempt: 1
I20260620 06:06:34.908406 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.908068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53316)
0620 06:06:34.908254 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.908323 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260620 06:06:34.909433 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.909186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53318)
0620 06:06:34.909311 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.909362 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260620 06:06:34.909698 25884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:45443: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
W20260620 06:06:34.975447 25891 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:06:34.975778 25891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:34.975803 25891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:34.975896 25891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260620 06:06:34.977883 25884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } attempt: 2
I20260620 06:06:34.978847 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.978527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53324)
0620 06:06:34.978701 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.978769 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 06:06:34.979599 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:34.979401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53336)
0620 06:06:34.979493 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:34.979538 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:06:34.979743 25884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:45443: Network error: Client connection negotiation failed: client connection to 127.3.115.61:45443: connect: Connection refused (error 111)
W20260620 06:06:34.980863 25891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:34.980921 25891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:34.980942 25891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:34.980997 25891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:34.981050 25891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:34.986872 25891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:37159,127.3.115.61:45443,127.3.115.60:37599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab
--principal=kudu/127.3.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:45443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:34.988095 25891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:34.989297 25891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:34.998749 25897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:34.998749 25899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:34.998747 25896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:34.999218 25891 server_base.cc:1034] running on GCE node
I20260620 06:06:34.999938 25891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:35.001282 25891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:35.002530 25891 hybrid_clock.cc:648] HybridClock initialized: now 1781935595002509 us; error 40 us; skew 500 ppm
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:35.006024 25891 init.cc:377] Logged in from keytab as kudu/127.3.115.61@KRBTEST.COM (short username kudu)
I20260620 06:06:35.007053 25891 webserver.cc:460] Webserver started at http://127.3.115.61:46801/ using document root <none> and password file <none>
I20260620 06:06:35.007666 25891 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:35.007733 25891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:35.007925 25891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:35.010375 25891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data/instance:
uuid: "f1a8d69ea0b242179912331925994391"
format_stamp: "Formatted at 2026-06-20 06:06:35 on dist-test-slave-4h14"
server_key: "778e6be87bc794ac0bb173ae1e1e5ce2"
server_key_iv: "f472e21b423e3f857bc46efa980b22da"
server_key_version: "encryptionkey@0"
I20260620 06:06:35.010987 25891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal/instance:
uuid: "f1a8d69ea0b242179912331925994391"
format_stamp: "Formatted at 2026-06-20 06:06:35 on dist-test-slave-4h14"
server_key: "778e6be87bc794ac0bb173ae1e1e5ce2"
server_key_iv: "f472e21b423e3f857bc46efa980b22da"
server_key_version: "encryptionkey@0"
I20260620 06:06:35.015231 25891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:35.018122 25906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:35.019551 25891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:35.019668 25891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal
uuid: "f1a8d69ea0b242179912331925994391"
format_stamp: "Formatted at 2026-06-20 06:06:35 on dist-test-slave-4h14"
server_key: "778e6be87bc794ac0bb173ae1e1e5ce2"
server_key_iv: "f472e21b423e3f857bc46efa980b22da"
server_key_version: "encryptionkey@0"
I20260620 06:06:35.019768 25891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:35.035571 25891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:35.039398 25891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:35.039577 25891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:35.047073 25891 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.61:45443
I20260620 06:06:35.047401 25958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.61:45443 every 8 connection(s)
I20260620 06:06:35.048161 25891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
I20260620 06:06:35.051200 25959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:35.055120 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25891
I20260620 06:06:35.055241 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-1/wal/instance
I20260620 06:06:35.055533 3532 external_mini_cluster.cc:1428] Setting key 5da441c251edbe86219b5984343476c8
I20260620 06:06:35.055215 25959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.115.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.60@KRBTEST.COM" created.
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:35.070902 25884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } attempt: 3
I20260620 06:06:35.073025 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.057321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.3.115.62:37159)
0620 06:06:35.057503 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.057510 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.058373 (+ 863us) server_negotiation.cc:407] Connection header received
0620 06:06:35.059455 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.059460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.059532 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.059649 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.060914 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.061808 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.063038 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.063467 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.063597 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.067293 (+ 3696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.067296 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.067298 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.067328 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.069844 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.070695 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.070699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.070700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.070786 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.071296 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.071302 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.071304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.071597 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.071712 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.072594 (+ 882us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.072854 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 06:06:35.074272 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.057699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37159 (local address 127.0.0.1:45130)
0620 06:06:35.058197 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.058252 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.059170 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.059812 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.059820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.060180 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.060752 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.060768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.061962 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.061965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.062896 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.062906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.063647 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.063650 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.063818 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.064523 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.064550 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.067023 (+ 2473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.070038 (+ 3015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.070080 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.070105 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.070571 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.070901 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.070904 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.070906 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.071128 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.071785 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.071796 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.072235 (+ 439us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.072542 (+ 307us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.073037 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":146,"threads_started":1}
I20260620 06:06:35.076053 25959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab.
Entry for principal kudu/127.3.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab.
I20260620 06:06:35.083820 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:37599
--webserver_interface=127.3.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab
--principal=kudu/127.3.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:37159,127.3.115.61:45443,127.3.115.60:37599
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:06:35.084168 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.071420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53346)
0620 06:06:35.071617 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.071634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.071715 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.072368 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.072371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.072567 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.072728 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.072736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.074042 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.074048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.074791 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.074801 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.075655 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.075658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.075766 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.076307 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.076322 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.078992 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.081529 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.081533 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.081536 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.081813 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.082207 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.082209 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.082210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.082332 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.083202 (+ 870us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.083205 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.083330 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.083463 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.083940 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116}
I20260620 06:06:35.084489 25964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.071522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53346 (local address 127.3.115.61:45443)
0620 06:06:35.071945 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.071955 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.071977 (+ 22us) server_negotiation.cc:407] Connection header received
0620 06:06:35.072013 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.072018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.072204 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.072363 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.072872 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.073918 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.074920 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.075529 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.075659 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.079137 (+ 3478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.079151 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.079169 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.079199 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.081397 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.081929 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.081937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.081939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.082100 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.082450 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.082454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.082455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.082959 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.083222 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.083597 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.083780 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":134,"threads_started":1}
I20260620 06:06:35.086359 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.076629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53348)
0620 06:06:35.076880 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.076899 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.076977 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.077263 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.077266 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.077426 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.077506 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.077511 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.078421 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.078424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.079178 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.079188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.079725 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.079728 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.079817 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.080241 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.080253 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.081768 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.084555 (+ 2787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.084562 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.084567 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.085029 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.085356 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.085361 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.085365 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.085454 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.085840 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.085843 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.085924 (+ 81us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.086037 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.086186 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260620 06:06:35.086530 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.076764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53348 (local address 127.3.115.61:45443)
0620 06:06:35.077062 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.077070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.077081 (+ 11us) server_negotiation.cc:407] Connection header received
0620 06:06:35.077114 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.077116 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.077147 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.077242 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.077620 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.078285 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.079303 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.079614 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.079729 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.081942 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.081951 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.081953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.081983 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.084449 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.085155 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.085160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.085165 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.085263 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.085548 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.085555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.085558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.085740 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.085849 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.086093 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.086299 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":231,"thread_start_us":137,"threads_started":1}
I20260620 06:06:35.086840 25884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:35.087106 25959 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:35.087823 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.087339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35918)
0620 06:06:35.087715 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.087761 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298}
I20260620 06:06:35.087869 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.087540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35930)
0620 06:06:35.087688 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.087784 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:06:35.088052 25909 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
I20260620 06:06:35.088708 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.088378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35944)
0620 06:06:35.088552 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.088602 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260620 06:06:35.088925 25884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:37599: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
I20260620 06:06:35.089069 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.088783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35960)
0620 06:06:35.088918 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.088981 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 06:06:35.089198 25959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:37599: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
I20260620 06:06:35.120091 25884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } attempt: 1
I20260620 06:06:35.121351 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.120921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35964)
0620 06:06:35.121180 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.121260 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
I20260620 06:06:35.122210 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.121959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35970)
0620 06:06:35.122087 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.122147 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 06:06:35.122396 25884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:37599: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
I20260620 06:06:35.138481 25959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } attempt: 1
I20260620 06:06:35.139451 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.139137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35982)
0620 06:06:35.139304 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.139370 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260620 06:06:35.140247 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.140032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:35992)
0620 06:06:35.140145 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.140193 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260620 06:06:35.140465 25959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:37599: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
I20260620 06:06:35.197584 25884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } attempt: 2
I20260620 06:06:35.198468 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.198170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36008)
0620 06:06:35.198323 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.198387 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260620 06:06:35.199245 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.199026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36020)
0620 06:06:35.199145 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.199194 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:06:35.199373 25884 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:37599: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
W20260620 06:06:35.202347 25966 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:06:35.202744 25966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:35.202807 25966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:35.202903 25966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:35.207197 25966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:35.207304 25966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:35.207342 25966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:35.207378 25966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:35.207422 25966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:35.208631 25959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } attempt: 2
I20260620 06:06:35.209546 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.209249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36032)
0620 06:06:35.209397 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.209471 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 06:06:35.210291 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.210070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36046)
0620 06:06:35.210178 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.210233 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 06:06:35.210419 25959 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:37599: Network error: Client connection negotiation failed: client connection to 127.3.115.60:37599: connect: Connection refused (error 111)
I20260620 06:06:35.213696 25966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:37159,127.3.115.61:45443,127.3.115.60:37599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab
--principal=kudu/127.3.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:37599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:35.214885 25966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:35.215989 25966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:35.224871 25972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:35.224864 25975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:35.224864 25973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:35.225265 25966 server_base.cc:1034] running on GCE node
I20260620 06:06:35.225862 25966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:35.227147 25966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:35.228335 25966 hybrid_clock.cc:648] HybridClock initialized: now 1781935595228324 us; error 35 us; skew 500 ppm
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:35.231329 25966 init.cc:377] Logged in from keytab as kudu/127.3.115.60@KRBTEST.COM (short username kudu)
I20260620 06:06:35.232367 25966 webserver.cc:460] Webserver started at http://127.3.115.60:46785/ using document root <none> and password file <none>
I20260620 06:06:35.233047 25966 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:35.233156 25966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:35.233412 25966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:35.235621 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data/instance:
uuid: "19967f35fad74c1f961d349e2328572f"
format_stamp: "Formatted at 2026-06-20 06:06:35 on dist-test-slave-4h14"
server_key: "d9e1e1677a359e2d5a730e8dbb67a3d9"
server_key_iv: "b955e5aec718a092be80b0831860c388"
server_key_version: "encryptionkey@0"
I20260620 06:06:35.236241 25966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal/instance:
uuid: "19967f35fad74c1f961d349e2328572f"
format_stamp: "Formatted at 2026-06-20 06:06:35 on dist-test-slave-4h14"
server_key: "d9e1e1677a359e2d5a730e8dbb67a3d9"
server_key_iv: "b955e5aec718a092be80b0831860c388"
server_key_version: "encryptionkey@0"
I20260620 06:06:35.240432 25966 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260620 06:06:35.243294 25982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:35.244901 25966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260620 06:06:35.245066 25966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal
uuid: "19967f35fad74c1f961d349e2328572f"
format_stamp: "Formatted at 2026-06-20 06:06:35 on dist-test-slave-4h14"
server_key: "d9e1e1677a359e2d5a730e8dbb67a3d9"
server_key_iv: "b955e5aec718a092be80b0831860c388"
server_key_version: "encryptionkey@0"
I20260620 06:06:35.245227 25966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:35.276427 25966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:35.280258 25966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:35.280494 25966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:35.290321 25966 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.60:37599
I20260620 06:06:35.290614 26034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.60:37599 every 8 connection(s)
I20260620 06:06:35.291500 25966 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
I20260620 06:06:35.292831 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 25966
I20260620 06:06:35.292971 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781935311537307-3532-0/minicluster-data/master-2/wal/instance
I20260620 06:06:35.293237 3532 external_mini_cluster.cc:1428] Setting key f3cbcb4d501fb407705924a7914d89f3
I20260620 06:06:35.295001 26035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:35.298542 25884 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } attempt: 3
I20260620 06:06:35.299907 26035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:35.301615 25959 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } attempt: 3
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:35.311340 25887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.299053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36072)
0620 06:06:35.299272 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.299287 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.299368 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.299849 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.299854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.300062 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.300164 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.300170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.301307 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.301311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.302075 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.302089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.302679 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.302684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.302801 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.303271 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.303288 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.304798 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.308957 (+ 4159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.308963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.308965 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.309267 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.309720 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.309724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.309725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.309823 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.310763 (+ 940us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.310768 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.310885 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.311009 (+ 124us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.311167 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20260620 06:06:35.312546 26039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.299092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36072 (local address 127.3.115.60:37599)
0620 06:06:35.299632 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.299639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.299651 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:35.299680 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.299682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.299715 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.299818 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.300320 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.301180 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.302218 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.302547 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.303024 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.304999 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.305006 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.305023 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.305062 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.308815 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.309373 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.309381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.309388 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.309563 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.309933 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.309937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.309940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.310589 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.310716 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.311183 (+ 467us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.311501 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":457,"thread_start_us":113,"threads_started":1}
I20260620 06:06:35.312546 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.294710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36062 (local address 127.3.115.60:37599)
0620 06:06:35.295142 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.295162 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.295205 (+ 43us) server_negotiation.cc:407] Connection header received
0620 06:06:35.296063 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.296086 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.296516 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.296921 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.297604 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.299147 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.300399 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.301189 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.301356 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.305691 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.305696 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.305702 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.305735 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.308607 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.309780 (+ 1173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.309783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.309785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.309905 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.310228 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.310234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.310238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.310598 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.311889 (+ 1291us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.311973 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.312092 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":67,"server-negotiator.queue_time_us":274,"spinlock_wait_cycles":4992,"thread_start_us":119,"threads_started":1}
I20260620 06:06:35.314402 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.301995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:37159 (local address 127.0.0.1:45132)
0620 06:06:35.303344 (+ 1349us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.303402 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.303551 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.303928 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.303933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.304421 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.304570 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.304577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.305631 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.305634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.306615 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.306623 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.307396 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.307400 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.307545 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.308127 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.308149 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.310333 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.312524 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.312532 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.312548 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.312878 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.313210 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.313213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.313217 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.313306 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.313790 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.313796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.313923 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.314063 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.314238 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1176,"thread_start_us":180,"threads_started":1}
I20260620 06:06:35.314443 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.302877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45132 (local address 127.3.115.62:37159)
0620 06:06:35.303042 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.303047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.303468 (+ 421us) server_negotiation.cc:407] Connection header received
0620 06:06:35.303657 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.303660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.303707 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.303763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.304740 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.305491 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.306737 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.307129 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.307250 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.310487 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.310491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.310494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.310522 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.312366 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.313013 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.313016 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.313018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.313106 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.313405 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.313407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.313408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.313660 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.313782 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.314162 (+ 380us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.314288 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 06:06:35.315771 26035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:35.315840 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.302068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36082)
0620 06:06:35.302395 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.302410 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.302491 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.303325 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.303328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.303535 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.303675 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.303682 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.304768 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.304771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.305630 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.305640 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.306205 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.306209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.306294 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.306818 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.306852 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.308280 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.310336 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.310340 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.310342 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.310589 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.310881 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.310887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.310891 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.311002 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.315316 (+ 4314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.315322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.315445 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.315540 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.315682 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246}
I20260620 06:06:35.316497 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.302653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36082 (local address 127.3.115.60:37599)
0620 06:06:35.303065 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.303074 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.303089 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:35.303125 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.303131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.303175 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.303781 (+ 606us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.303877 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.304617 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.305760 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.306077 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.306501 (+ 424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.308421 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.308424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.308427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.308457 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.310216 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.310713 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.310716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.310718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.310793 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.314954 (+ 4161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.314957 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.314959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.315176 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.316128 (+ 952us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.316207 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.316323 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":90,"threads_started":1}
I20260620 06:06:35.317669 25884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07: Bootstrap starting.
I20260620 06:06:35.319392 25959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: Bootstrap starting.
I20260620 06:06:35.320484 25884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:35.321435 25884 log.cc:824] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:35.322067 25959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: Neither blocks nor log segments found. Creating new log.
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:35.322845 25959 log.cc:824] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:35.325439 25884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07: No bootstrap required, opened a new log
I20260620 06:06:35.326062 25959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: No bootstrap required, opened a new log
I20260620 06:06:35.327275 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.316436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.3.115.61:45443)
0620 06:06:35.316548 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.316554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.316572 (+ 18us) server_negotiation.cc:407] Connection header received
0620 06:06:35.316627 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.316630 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.316732 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.316819 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.317525 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.318365 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.319583 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.319944 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.320013 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.323219 (+ 3206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.323222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.323224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.323247 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.325328 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.325858 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.325862 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.325863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.325949 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.326285 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.326289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.326290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.326540 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.326615 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.326975 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.327116 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 06:06:35.327551 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.315446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.3.115.62:37159)
0620 06:06:35.315543 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.315550 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.315712 (+ 162us) server_negotiation.cc:407] Connection header received
0620 06:06:35.315893 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.315896 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.315941 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.315989 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.317701 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.318404 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.319604 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.319935 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.320013 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.323205 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.323212 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.323218 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.323252 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.325126 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.325684 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.325688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.325690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.325774 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.326099 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.326102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.326104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.326331 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.326495 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.327267 (+ 772us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.327404 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 06:06:35.327790 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.316238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:45443 (local address 127.0.0.1:53360)
0620 06:06:35.316447 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.316463 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.316534 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.317038 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.317041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.317231 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.317376 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.317385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.318547 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.318551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.319408 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.319418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.320273 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.320276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.320370 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.320899 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.320916 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.323026 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.325469 (+ 2443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.325473 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.325475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.325735 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.326066 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.326070 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.326086 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.326178 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.326691 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.326694 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.326793 (+ 99us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.327465 (+ 672us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.327627 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146}
I20260620 06:06:35.328488 26035 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:35.330884 25959 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.331173 25959 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1a8d69ea0b242179912331925994391, State: Initialized, Role: FOLLOWER
I20260620 06:06:35.331460 25884 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.331547 25959 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.331815 25884 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2446777aabc44658e70c7f91d7e7e07, State: Initialized, Role: FOLLOWER
I20260620 06:06:35.332198 25884 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.333591 25959 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:35.333822 25884 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935595, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:06:35 dist-test-slave-4h14 krb5kdc[25803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:06:35.335096 26045 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.335673 26045 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.338240 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.328994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:37599 (local address 127.0.0.1:36086)
0620 06:06:35.329180 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:35.329193 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:35.329273 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:35.329563 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:35.329565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:35.329752 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:35.329842 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.329848 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.330840 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.330843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.331598 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:35.331608 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.332271 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.332274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:35.332340 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.332879 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:35.332895 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:35.334513 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:35.336628 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.336631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.336633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.336900 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.337197 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:35.337199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:35.337201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:35.337283 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:35.337718 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:35.337721 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:35.337798 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 06:06:35.337891 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 06:06:35.338034 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142}
I20260620 06:06:35.338687 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.329007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.3.115.60:37599)
0620 06:06:35.329120 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.329124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.329342 (+ 218us) server_negotiation.cc:407] Connection header received
0620 06:06:35.329367 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.329369 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.329406 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.329463 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.329962 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.330724 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.331722 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.332049 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.332156 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.334684 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.334689 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.334694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.334727 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.336502 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.337007 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.337011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.337013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.337103 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.337371 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.337374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.337376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.337590 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.337686 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.338437 (+ 751us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.338567 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 06:06:35.339613 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.339848 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.340173 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.328920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.3.115.61:45443)
0620 06:06:35.329024 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.329029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.329328 (+ 299us) server_negotiation.cc:407] Connection header received
0620 06:06:35.329366 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.329369 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.329410 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.329473 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.330325 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.331061 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.332148 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.332454 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.332566 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.336266 (+ 3700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.336270 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.336273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.336299 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.338060 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.338610 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.338613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.338615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.338701 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.339033 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.339048 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.339050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.339274 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.339466 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.339865 (+ 399us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.340001 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 06:06:35.341977 26035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: Bootstrap starting.
W20260620 06:06:35.343938 26067 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:35.344041 26067 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:35.345049 26035 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:35.346068 26035 log.cc:824] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:35.343827 25816 master_runner.cc:186] Error getting master registration for 127.3.115.60:37599: OK, instance_id { permanent_uuid: "19967f35fad74c1f961d349e2328572f" instance_seqno: 1781935595287744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:06:35.348099 25816 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 06:06:35.348137 26065 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:35.348232 26065 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:35.348438 26035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: No bootstrap required, opened a new log
I20260620 06:06:35.348568 25891 master_runner.cc:186] Error getting master registration for 127.3.115.60:37599: OK, instance_id { permanent_uuid: "19967f35fad74c1f961d349e2328572f" instance_seqno: 1781935595287744 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:06:35.352254 25891 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:06:35.353387 26035 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.353618 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:35.353680 26035 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19967f35fad74c1f961d349e2328572f, State: Initialized, Role: FOLLOWER
I20260620 06:06:35.354038 26035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.354643 26071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.354866 26071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.355041 25858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d2446777aabc44658e70c7f91d7e7e07"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45144
I20260620 06:06:35.355321 25858 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 06:06:35.355441 25858 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:35.355664 26035 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:35.357048 25858 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.358747 25858 leader_election.cc:288] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [CANDIDATE]: Term 1 election: Requested vote from peers f1a8d69ea0b242179912331925994391 (127.3.115.61:45443), 19967f35fad74c1f961d349e2328572f (127.3.115.60:37599)
I20260620 06:06:35.358752 25933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2446777aabc44658e70c7f91d7e7e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f1a8d69ea0b242179912331925994391"
I20260620 06:06:35.358992 25933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:35.359541 26009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d2446777aabc44658e70c7f91d7e7e07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "19967f35fad74c1f961d349e2328572f"
I20260620 06:06:35.359742 26009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:35.360533 25933 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2446777aabc44658e70c7f91d7e7e07 in term 1.
I20260620 06:06:35.361013 25835 leader_election.cc:302] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d2446777aabc44658e70c7f91d7e7e07, f1a8d69ea0b242179912331925994391; no voters:
I20260620 06:06:35.361318 26045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:35.361527 26045 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [term 1 LEADER]: Becoming Leader. State: Replica: d2446777aabc44658e70c7f91d7e7e07, State: Running, Role: LEADER
I20260620 06:06:35.361871 26045 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } }
I20260620 06:06:35.362735 26009 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2446777aabc44658e70c7f91d7e7e07 in term 1.
I20260620 06:06:35.363768 26045 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2446777aabc44658e70c7f91d7e7e07. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.363902 26045 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: This master's current role is: LEADER
W20260620 06:06:35.365751 26082 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:35.365876 26082 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:35.367686 26090 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:35.368737 26090 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:35.368738 25966 master_runner.cc:263] No leader master found from master 19967f35fad74c1f961d349e2328572f
I20260620 06:06:35.368800 25966 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:06:35.375123 26009 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [term 1 FOLLOWER]: Refusing update from remote peer d2446777aabc44658e70c7f91d7e7e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:35.375933 25933 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [term 1 FOLLOWER]: Refusing update from remote peer d2446777aabc44658e70c7f91d7e7e07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:35.376780 26045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:35.377223 26045 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:35.384195 26071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2446777aabc44658e70c7f91d7e7e07. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.384371 26071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.386112 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2446777aabc44658e70c7f91d7e7e07. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.386269 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.393666 26087 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.393838 26087 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:35.393967 26091 mvcc.cc:205] Tried to move back new op lower bound from 7298808198648594432 to 7298808198605938688. Current Snapshot: MvccSnapshot[applied={T|T < 7298808198648594432}]
I20260620 06:06:35.395134 26044 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.395273 26044 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.396536 26045 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.396708 26045 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:35.397957 26071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d2446777aabc44658e70c7f91d7e7e07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2446777aabc44658e70c7f91d7e7e07" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 37159 } } peers { permanent_uuid: "f1a8d69ea0b242179912331925994391" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 45443 } } peers { permanent_uuid: "19967f35fad74c1f961d349e2328572f" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 37599 } } }
I20260620 06:06:35.398109 26071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:35.398942 26097 mvcc.cc:205] Tried to move back new op lower bound from 7298808198648594432 to 7298808198605938688. Current Snapshot: MvccSnapshot[applied={T|T < 7298808198648594432}]
I20260620 06:06:35.399176 26095 mvcc.cc:205] Tried to move back new op lower bound from 7298808198648594432 to 7298808198605938688. Current Snapshot: MvccSnapshot[applied={T|T < 7298808198648594432}]
I20260620 06:06:35.403237 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.371971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53372 (local address 127.3.115.61:45443)
0620 06:06:35.372081 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.372087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.372930 (+ 843us) server_negotiation.cc:407] Connection header received
0620 06:06:35.373151 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.373155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.373207 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.373262 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.388809 (+ 15547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.389555 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.392433 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.392782 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.392866 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.398081 (+ 5215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.398087 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.398092 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.398125 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.399867 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.401489 (+ 1622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.401492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.401494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.401609 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.401928 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.401932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.401933 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.402166 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.402279 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.402950 (+ 671us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.403080 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 06:06:35.406466 26090 catalog_manager.cc:1347] Generated new cluster ID: ea26bf76a004420a813434f68d0c2cfc
I20260620 06:06:35.406558 26090 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:35.410612 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.371697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36090 (local address 127.3.115.60:37599)
0620 06:06:35.371809 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.371813 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.391079 (+ 19266us) server_negotiation.cc:407] Connection header received
0620 06:06:35.391279 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.391283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.391334 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.391385 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.393655 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.394352 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.396996 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.397293 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.397372 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.403928 (+ 6556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.403932 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.403934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.403961 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.405753 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.407259 (+ 1506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.407263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.407264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.407360 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.407691 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.407694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.407696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.407901 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.408013 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.410295 (+ 2282us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.410431 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:06:35.427310 25888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:35.371475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.3.115.62:37159)
0620 06:06:35.371600 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.371606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.402821 (+ 31215us) server_negotiation.cc:407] Connection header received
0620 06:06:35.406285 (+ 3464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.406289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.406354 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.406412 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.408295 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.409089 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.412253 (+ 3164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:35.412618 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:35.412733 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:35.423073 (+ 10340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:35.423077 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:35.423080 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:35.423114 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:35.425159 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.425773 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.425777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.425779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.425878 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:35.426219 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:35.426222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:35.426224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:35.426465 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:35.426585 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:35.426980 (+ 395us) server_negotiation.cc:299] Negotiation successful
0620 06:06:35.427128 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 06:06:35.435685 26090 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:35.437191 26090 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:35.467489 26090 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2446777aabc44658e70c7f91d7e7e07: Generated new TSK 0
I20260620 06:06:35.468501 26090 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:06:35.496289 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.492979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53374 (local address 127.3.115.61:45443)
0620 06:06:35.493113 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.493119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.495115 (+ 1996us) server_negotiation.cc:407] Connection header received
0620 06:06:35.495331 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.495335 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.495388 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.495444 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.496139 (+ 695us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53374: BlockingRecv error: recv got EOF from 127.0.0.1:53374 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 06:06:35.497802 26041 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.493787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.3.115.60:37599)
0620 06:06:35.493900 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.493905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.496841 (+ 2936us) server_negotiation.cc:407] Connection header received
0620 06:06:35.497019 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.497022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.497084 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.497135 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.497666 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36094: BlockingRecv error: recv got EOF from 127.0.0.1:36094 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260620 06:06:35.499343 25888 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.494487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45168 (local address 127.3.115.62:37159)
0620 06:06:35.494603 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.494607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.498367 (+ 3760us) server_negotiation.cc:407] Connection header received
0620 06:06:35.498575 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.498578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.498631 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.498685 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.499211 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45168: BlockingRecv error: recv got EOF from 127.0.0.1:45168 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260620 06:06:35.503741 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.502099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.3.115.61:45443)
0620 06:06:35.502217 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.502222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.502683 (+ 461us) server_negotiation.cc:407] Connection header received
0620 06:06:35.502914 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.502917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.502997 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.503051 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.503619 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53376: BlockingRecv error: recv got EOF from 127.0.0.1:53376 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 06:06:35.509282 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.507526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53390 (local address 127.3.115.61:45443)
0620 06:06:35.507643 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.507648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.508230 (+ 582us) server_negotiation.cc:407] Connection header received
0620 06:06:35.508439 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.508443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.508497 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.508549 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.509158 (+ 609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53390: BlockingRecv error: recv got EOF from 127.0.0.1:53390 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 06:06:35.515249 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.513520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.3.115.61:45443)
0620 06:06:35.513639 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.513643 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.514231 (+ 588us) server_negotiation.cc:407] Connection header received
0620 06:06:35.514434 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.514437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.514491 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.514544 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.515125 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53394: BlockingRecv error: recv got EOF from 127.0.0.1:53394 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 06:06:35.523146 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.521609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.3.115.61:45443)
0620 06:06:35.521785 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.521790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.522064 (+ 274us) server_negotiation.cc:407] Connection header received
0620 06:06:35.522283 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.522286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.522339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.522391 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.523010 (+ 619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53408: BlockingRecv error: recv got EOF from 127.0.0.1:53408 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260620 06:06:35.535708 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.534097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.3.115.61:45443)
0620 06:06:35.534246 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.534253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.534593 (+ 340us) server_negotiation.cc:407] Connection header received
0620 06:06:35.534829 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.534833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.534895 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.534954 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.535580 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53410: BlockingRecv error: recv got EOF from 127.0.0.1:53410 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260620 06:06:35.556999 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.554710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.3.115.61:45443)
0620 06:06:35.554858 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.554864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.555686 (+ 822us) server_negotiation.cc:407] Connection header received
0620 06:06:35.555951 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.555955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.556018 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.556089 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.556863 (+ 774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53424: BlockingRecv error: recv got EOF from 127.0.0.1:53424 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260620 06:06:35.592756 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.591584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53430 (local address 127.3.115.61:45443)
0620 06:06:35.591747 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.591751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.591839 (+ 88us) server_negotiation.cc:407] Connection header received
0620 06:06:35.591973 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.591975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.592026 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.592116 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.592571 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53430: BlockingRecv error: recv got EOF from 127.0.0.1:53430 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260620 06:06:35.660234 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.659048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53446 (local address 127.3.115.61:45443)
0620 06:06:35.659203 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.659207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.659321 (+ 114us) server_negotiation.cc:407] Connection header received
0620 06:06:35.659466 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.659469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.659516 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.659596 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.660105 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53446: BlockingRecv error: recv got EOF from 127.0.0.1:53446 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 06:06:35.792130 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:35.790820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53458 (local address 127.3.115.61:45443)
0620 06:06:35.791004 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:35.791009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:35.791167 (+ 158us) server_negotiation.cc:407] Connection header received
0620 06:06:35.791361 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:35.791364 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:35.791415 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:35.791508 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:35.792005 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53458: BlockingRecv error: recv got EOF from 127.0.0.1:53458 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260620 06:06:36.051772 25965 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:36.050539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53462 (local address 127.3.115.61:45443)
0620 06:06:36.050677 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:36.050683 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:36.050907 (+ 224us) server_negotiation.cc:407] Connection header received
0620 06:06:36.051025 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:36.051028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:36.051087 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:36.051188 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:36.051670 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53462: BlockingRecv error: recv got EOF from 127.0.0.1:53462 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 06:06:36.349684 26065 catalog_manager.cc:1259] Loaded cluster ID: ea26bf76a004420a813434f68d0c2cfc
I20260620 06:06:36.349772 26065 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: loading cluster ID for follower catalog manager: success
I20260620 06:06:36.351523 26065 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: acquiring CA information for follower catalog manager: success
I20260620 06:06:36.352772 26065 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f1a8d69ea0b242179912331925994391: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:06:36.367197 26082 catalog_manager.cc:1259] Loaded cluster ID: ea26bf76a004420a813434f68d0c2cfc
I20260620 06:06:36.367283 26082 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: loading cluster ID for follower catalog manager: success
I20260620 06:06:36.369056 26082 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: acquiring CA information for follower catalog manager: success
I20260620 06:06:36.370374 26082 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19967f35fad74c1f961d349e2328572f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:06:36.570242 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:36.566081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.3.115.61:45443)
0620 06:06:36.566355 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:36.566359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:36.566376 (+ 17us) server_negotiation.cc:407] Connection header received
0620 06:06:36.566504 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:36.566507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:36.566634 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:36.566735 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:36.566992 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.567577 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.568454 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.568766 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.568858 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:36.569239 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:36.569716 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:36.569825 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:36.569918 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 06:06:36.570008 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260620 06:06:36.577548 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:36.573693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.3.115.60:37599)
0620 06:06:36.573930 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:36.573935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:36.573950 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:36.574064 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:36.574067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:36.574186 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:36.574315 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:36.574584 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.575116 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.575978 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.576219 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.576337 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:36.576677 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:36.577135 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:36.577283 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:36.577319 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 06:06:36.577381 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":60,"threads_started":1}
I20260620 06:06:36.584972 26172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:36.580929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.3.115.60:37599)
0620 06:06:36.581182 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:36.581186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:36.581200 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:06:36.581406 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:36.581409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:36.581457 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:36.581523 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:36.581850 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.582674 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.583681 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.583976 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.584068 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:36.584460 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:36.584547 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:36.584628 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:36.584749 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 06:06:36.584808 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260620 06:06:36.585295 26166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:36.580872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53478 (local address 127.3.115.61:45443)
0620 06:06:36.581007 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:36.581013 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:36.581505 (+ 492us) server_negotiation.cc:407] Connection header received
0620 06:06:36.581695 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:36.581701 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:36.581764 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:36.581832 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:36.582200 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.582971 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.584048 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.584325 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.584441 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:36.584784 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:36.584862 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:36.584928 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:36.585087 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 06:06:36.585160 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:06:36.585783 26179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:36.580962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45172 (local address 127.3.115.62:37159)
0620 06:06:36.581347 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:36.581352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:36.581366 (+ 14us) server_negotiation.cc:407] Connection header received
0620 06:06:36.581459 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:36.581462 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:36.581632 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:36.581757 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:36.581998 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.582625 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.583849 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:36.584194 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:36.584293 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:36.584714 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:36.585102 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:36.585186 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:36.585579 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 06:06:36.585649 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":75,"threads_started":1}
I20260620 06:06:36.589321 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25816
I20260620 06:06:36.595863 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25891
I20260620 06:06:36.601898 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 25966
2026-06-20T06:06:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3150 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 06:06:36 dist-test-slave-4h14 krb5kdc[26183](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 06:06:36 dist-test-slave-4h14 krb5kdc[26183](info): set up 2 sockets
Jun 20 06:06:36 dist-test-slave-4h14 krb5kdc[26183](info): commencing operation
krb5kdc: starting...
W20260620 06:06:37.645277 3532 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/test-user.keytab.
Jun 20 06:06:37 dist-test-slave-4h14 krb5kdc[26183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T06:06:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T06:06:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.115.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
Entry for principal kudu/127.3.115.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab.
I20260620 06:06:37.772127 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43365
--webserver_interface=127.3.115.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43365,127.3.115.61:44793,127.3.115.60:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 06:06:37.887529 26196 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:06:37.887917 26196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:37.887975 26196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:37.888059 26196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:37.892169 26196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:37.892294 26196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:37.892334 26196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:37.892376 26196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:37.892410 26196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:37.897945 26196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43365,127.3.115.61:44793,127.3.115.60:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.62.keytab
--principal=kudu/127.3.115.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.62:43365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.115.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:37.899201 26196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:37.900326 26196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 06:06:37.910084 26196 server_base.cc:1034] running on GCE node
W20260620 06:06:37.910105 26201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:37.910060 26202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:37.910101 26204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:37.910972 26196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:37.912411 26196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:37.913645 26196 hybrid_clock.cc:648] HybridClock initialized: now 1781935597913628 us; error 46 us; skew 500 ppm
Jun 20 06:06:37 dist-test-slave-4h14 krb5kdc[26183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:37.917286 26196 init.cc:377] Logged in from keytab as kudu/127.3.115.62@KRBTEST.COM (short username kudu)
I20260620 06:06:37.918428 26196 webserver.cc:460] Webserver started at http://127.3.115.62:45659/ using document root <none> and password file <none>
I20260620 06:06:37.919085 26196 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:37.919163 26196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:37.919420 26196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:37.921794 26196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data/instance:
uuid: "b35ad3c107624560a57502ed402c34c2"
format_stamp: "Formatted at 2026-06-20 06:06:37 on dist-test-slave-4h14"
server_key: "2022ffdfbe96c3b7e05bf2be064c4651"
server_key_iv: "47814c1d4249aa7d822c355cbd8a2957"
server_key_version: "encryptionkey@0"
I20260620 06:06:37.922435 26196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal/instance:
uuid: "b35ad3c107624560a57502ed402c34c2"
format_stamp: "Formatted at 2026-06-20 06:06:37 on dist-test-slave-4h14"
server_key: "2022ffdfbe96c3b7e05bf2be064c4651"
server_key_iv: "47814c1d4249aa7d822c355cbd8a2957"
server_key_version: "encryptionkey@0"
I20260620 06:06:37.926867 26196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:37.930073 26211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:37.931676 26196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260620 06:06:37.931872 26196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal
uuid: "b35ad3c107624560a57502ed402c34c2"
format_stamp: "Formatted at 2026-06-20 06:06:37 on dist-test-slave-4h14"
server_key: "2022ffdfbe96c3b7e05bf2be064c4651"
server_key_iv: "47814c1d4249aa7d822c355cbd8a2957"
server_key_version: "encryptionkey@0"
I20260620 06:06:37.932035 26196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:37.949877 26196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:37.953176 26196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:37.953406 26196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:37.960863 26196 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.62:43365
I20260620 06:06:37.961176 26263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.62:43365 every 8 connection(s)
I20260620 06:06:37.961969 26196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/data/info.pb
I20260620 06:06:37.964973 26264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:37.969203 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 26196
I20260620 06:06:37.968964 26264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:37.969362 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-0/wal/instance
I20260620 06:06:37.969683 3532 external_mini_cluster.cc:1428] Setting key 0a08d5f594bce99dca71d8942c666c7b
WARNING: no policy specified for kudu/127.3.115.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.61@KRBTEST.COM" created.
Jun 20 06:06:37 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:37.989081 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:37.971772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34218 (local address 127.3.115.62:43365)
0620 06:06:37.972385 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:37.972395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:37.972459 (+ 64us) server_negotiation.cc:407] Connection header received
0620 06:06:37.974082 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:37.974086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:37.974370 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:37.974553 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:37.975824 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:37.976835 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:37.978434 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:37.979013 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:37.979149 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:37.983147 (+ 3998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:37.983159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:37.983181 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:37.983219 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:37.985286 (+ 2067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:37.985913 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:37.985920 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:37.985923 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:37.986100 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:37.986469 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:37.986472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:37.986474 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:37.986912 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:37.987069 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:37.987518 (+ 449us) server_negotiation.cc:299] Negotiation successful
0620 06:06:37.987775 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"spinlock_wait_cycles":14592,"thread_start_us":181,"threads_started":1}
I20260620 06:06:37.989081 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:37.971772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43365 (local address 127.0.0.1:34218)
0620 06:06:37.972296 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:37.972351 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:37.973727 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:37.974629 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:37.974637 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:37.975165 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:37.975617 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:37.975643 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:37.976967 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:37.976972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:37.978294 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:37.978307 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:37.979174 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:37.979177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:37.979274 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:37.979962 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:37.979980 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:37.982942 (+ 2962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:37.985480 (+ 2538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:37.985490 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:37.985504 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:37.985762 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:37.986215 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:37.986219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:37.986222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:37.986349 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:37.987035 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:37.987042 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:37.987199 (+ 157us) client_negotiation.cc:769] Sending connection context
0620 06:06:37.987396 (+ 197us) client_negotiation.cc:240] Negotiation successful
0620 06:06:37.987861 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":121,"thread_start_us":156,"threads_started":1}
I20260620 06:06:37.991743 26264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:37.992590 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:37.992292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35232)
0620 06:06:37.992420 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:37.992518 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 06:06:37.992877 26214 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
I20260620 06:06:37.993899 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:37.993676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35240)
0620 06:06:37.993787 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:37.993839 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 06:06:37.994084 26264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:44793: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab.
Entry for principal kudu/127.3.115.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab.
I20260620 06:06:38.000078 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:44793
--webserver_interface=127.3.115.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab
--principal=kudu/127.3.115.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43365,127.3.115.61:44793,127.3.115.60:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:06:38.043366 26264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } attempt: 1
I20260620 06:06:38.044342 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.044026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35256)
0620 06:06:38.044183 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.044262 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260620 06:06:38.045174 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.044954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35268)
0620 06:06:38.045069 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.045121 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:06:38.045303 26264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:44793: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
I20260620 06:06:38.113479 26264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } attempt: 2
I20260620 06:06:38.114403 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.114095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35272)
0620 06:06:38.114256 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.114321 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 06:06:38.115185 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.114948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35278)
0620 06:06:38.115090 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.115133 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 06:06:38.115311 26264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.61:44793: Network error: Client connection negotiation failed: client connection to 127.3.115.61:44793: connect: Connection refused (error 111)
W20260620 06:06:38.117659 26271 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 06:06:38.118023 26271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:38.118088 26271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:38.118170 26271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:38.122519 26271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:38.122620 26271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:38.122656 26271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:38.122713 26271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:38.122751 26271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:38.128537 26271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43365,127.3.115.61:44793,127.3.115.60:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.61.keytab
--principal=kudu/127.3.115.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.61:44793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.115.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:38.129721 26271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:38.130748 26271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:38.139729 26276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:38.139734 26277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:38.139809 26279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:38.140512 26271 server_base.cc:1034] running on GCE node
I20260620 06:06:38.141150 26271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:38.142324 26271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:38.143535 26271 hybrid_clock.cc:648] HybridClock initialized: now 1781935598143523 us; error 34 us; skew 500 ppm
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:38.146612 26271 init.cc:377] Logged in from keytab as kudu/127.3.115.61@KRBTEST.COM (short username kudu)
I20260620 06:06:38.147544 26271 webserver.cc:460] Webserver started at http://127.3.115.61:39027/ using document root <none> and password file <none>
I20260620 06:06:38.148103 26271 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:38.148165 26271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:38.148346 26271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:38.150593 26271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data/instance:
uuid: "4c3347eeabcc423f94272f28f68dae92"
format_stamp: "Formatted at 2026-06-20 06:06:38 on dist-test-slave-4h14"
server_key: "626f4de7dc350f9fa5a7f9b2a2721f94"
server_key_iv: "d2f1bb0fc5562babf567ef2b9db9adda"
server_key_version: "encryptionkey@0"
I20260620 06:06:38.151160 26271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal/instance:
uuid: "4c3347eeabcc423f94272f28f68dae92"
format_stamp: "Formatted at 2026-06-20 06:06:38 on dist-test-slave-4h14"
server_key: "626f4de7dc350f9fa5a7f9b2a2721f94"
server_key_iv: "d2f1bb0fc5562babf567ef2b9db9adda"
server_key_version: "encryptionkey@0"
I20260620 06:06:38.155232 26271 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260620 06:06:38.158066 26286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:38.159684 26271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 06:06:38.159871 26271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal
uuid: "4c3347eeabcc423f94272f28f68dae92"
format_stamp: "Formatted at 2026-06-20 06:06:38 on dist-test-slave-4h14"
server_key: "626f4de7dc350f9fa5a7f9b2a2721f94"
server_key_iv: "d2f1bb0fc5562babf567ef2b9db9adda"
server_key_version: "encryptionkey@0"
I20260620 06:06:38.160017 26271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:38.175827 26271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:38.179085 26271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:38.179313 26271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:38.186868 26271 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.61:44793
I20260620 06:06:38.187209 26338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.61:44793 every 8 connection(s)
I20260620 06:06:38.187942 26271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/data/info.pb
I20260620 06:06:38.190938 26339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:38.195070 26339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:38.197117 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 26271
I20260620 06:06:38.197299 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-1/wal/instance
I20260620 06:06:38.197628 3532 external_mini_cluster.cc:1428] Setting key 484567cdf61f25b58f8dd398885835be
WARNING: no policy specified for kudu/127.3.115.60@KRBTEST.COM; defaulting to no policy
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:38.206506 26264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.115.60@KRBTEST.COM" created.
I20260620 06:06:38.211850 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.197224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.3.115.62:43365)
0620 06:06:38.197398 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.197404 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.198412 (+ 1008us) server_negotiation.cc:407] Connection header received
0620 06:06:38.199456 (+ 1044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.199462 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.199544 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.199658 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.200920 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.201826 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.203020 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.203436 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.203573 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.206576 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.206579 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.206581 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.206610 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.209300 (+ 2690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.210004 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.210007 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.210009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.210104 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.210596 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.210601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.210602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.210804 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.210917 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.211556 (+ 639us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.211694 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:06:38.213516 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.197689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43365 (local address 127.0.0.1:34228)
0620 06:06:38.198221 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.198280 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.199225 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.199822 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.199829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.200193 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.200762 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.200778 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.201953 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.201955 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.202880 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.202890 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.203642 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.203645 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.203794 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.204530 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.204572 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.206315 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.209439 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.209450 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.209471 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.209893 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.210212 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.210218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.210223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.210443 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.210929 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.210938 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.211220 (+ 282us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.212095 (+ 875us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.212495 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":150,"threads_started":1}
I20260620 06:06:38.215171 26339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:38.218169 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.206986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35290)
0620 06:06:38.207146 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.207160 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.207267 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.207990 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.207994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.208264 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.208419 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.208425 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.209648 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.209653 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.210375 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.210387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.211302 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.211306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.211433 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.211887 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.211904 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.213581 (+ 1677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.216197 (+ 2616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.216203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.216205 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.216545 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.216934 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.216938 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.216940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.217045 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.217625 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.217628 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.217735 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.217848 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.217992 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260620 06:06:38.218396 26343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.207136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35290 (local address 127.3.115.61:44793)
0620 06:06:38.207533 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.207543 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.207564 (+ 21us) server_negotiation.cc:407] Connection header received
0620 06:06:38.207602 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.207607 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.207838 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.207959 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.208571 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.209508 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.210520 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.211167 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.211285 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.213778 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.213789 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.213809 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.213840 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.216058 (+ 2218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.216724 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.216732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.216734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.216828 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.217155 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.217160 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.217162 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.217512 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.217666 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.217977 (+ 311us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.218153 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":173,"threads_started":1}
I20260620 06:06:38.221086 26264 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:06:38.221902 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.221644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34302)
0620 06:06:38.221769 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.221823 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 06:06:38.222726 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.222493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34316)
0620 06:06:38.222633 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.222677 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260620 06:06:38.222934 26264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:46783: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
I20260620 06:06:38.225224 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.215735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35294 (local address 127.3.115.61:44793)
0620 06:06:38.216082 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.216091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.216104 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:38.216180 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.216183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.216226 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.216306 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.216723 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.217434 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.218450 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.218800 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.218911 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.221913 (+ 3002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.221917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.221918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.221938 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.223551 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.223975 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.223980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.223982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.224064 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.224328 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.224333 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.224335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.224542 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.224644 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.224907 (+ 263us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.225075 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
I20260620 06:06:38.225206 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.215735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35294)
0620 06:06:38.215991 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.216006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.216090 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.216323 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.216326 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.216499 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.216587 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.216592 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.217535 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.217541 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.218336 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.218346 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.218916 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.218923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.219045 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.219488 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.219497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.221740 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.223681 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.223685 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.223688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.223878 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.224152 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.224157 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.224159 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.224241 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.224641 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.224644 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.224751 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.224865 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.225016 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260620 06:06:38.226348 26339 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.115.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab.
Entry for principal kudu/127.3.115.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab.
I20260620 06:06:38.227113 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.226810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34320)
0620 06:06:38.226911 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.226999 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 06:06:38.227349 26290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
I20260620 06:06:38.227964 3532 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
/tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:46783
--webserver_interface=127.3.115.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab
--principal=kudu/127.3.115.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.115.20:45123
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.115.62:43365,127.3.115.61:44793,127.3.115.60:46783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 06:06:38.228345 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.228105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34332)
0620 06:06:38.228257 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.228294 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260620 06:06:38.228513 26339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:46783: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
I20260620 06:06:38.254132 26264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } attempt: 1
I20260620 06:06:38.255089 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.254732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34338)
0620 06:06:38.254906 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.254980 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 06:06:38.255880 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.255640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34350)
0620 06:06:38.255764 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.255819 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260620 06:06:38.256021 26264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:46783: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
I20260620 06:06:38.277871 26339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } attempt: 1
I20260620 06:06:38.278822 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.278498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34354)
0620 06:06:38.278663 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.278739 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260620 06:06:38.279685 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.279406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34358)
0620 06:06:38.279568 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.279626 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 06:06:38.279872 26339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:46783: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
I20260620 06:06:38.331210 26264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } attempt: 2
I20260620 06:06:38.332178 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.331808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34370)
0620 06:06:38.331988 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.332077 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 06:06:38.333057 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.332807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34376)
0620 06:06:38.332918 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.332981 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260620 06:06:38.333204 26264 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:46783: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
I20260620 06:06:38.348078 26339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } attempt: 2
I20260620 06:06:38.349054 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.348750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34390)
0620 06:06:38.348891 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.348972 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260620 06:06:38.349563 26347 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260620 06:06:38.349819 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.349596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34394)
0620 06:06:38.349702 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.349759 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 06:06:38.349929 26347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 06:06:38.350004 26347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 06:06:38.350107 26347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 06:06:38.350142 26339 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.115.60:46783: Network error: Client connection negotiation failed: client connection to 127.3.115.60:46783: connect: Connection refused (error 111)
W20260620 06:06:38.354568 26347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 06:06:38.354647 26347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 06:06:38.354668 26347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 06:06:38.354686 26347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 06:06:38.354702 26347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 06:06:38.360495 26347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.115.20:45123
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.115.62:43365,127.3.115.61:44793,127.3.115.60:46783
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/krb5kdc/kudu_127.3.115.60.keytab
--principal=kudu/127.3.115.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.115.60:46783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.115.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 20 Jun 2026 05:28:48 UTC on ccycloud.quasar-fixhms.root.comops.site
build id 4457020
I20260620 06:06:38.361721 26347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 06:06:38.362758 26347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 06:06:38.371899 26352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:38.371881 26355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 06:06:38.371881 26353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 06:06:38.372273 26347 server_base.cc:1034] running on GCE node
I20260620 06:06:38.373072 26347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 06:06:38.374404 26347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 06:06:38.375638 26347 hybrid_clock.cc:648] HybridClock initialized: now 1781935598375606 us; error 57 us; skew 500 ppm
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 06:06:38.378654 26347 init.cc:377] Logged in from keytab as kudu/127.3.115.60@KRBTEST.COM (short username kudu)
I20260620 06:06:38.379643 26347 webserver.cc:460] Webserver started at http://127.3.115.60:44699/ using document root <none> and password file <none>
I20260620 06:06:38.380262 26347 fs_manager.cc:359] Metadata directory not provided
I20260620 06:06:38.380363 26347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 06:06:38.380620 26347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 06:06:38.382848 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data/instance:
uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67"
format_stamp: "Formatted at 2026-06-20 06:06:38 on dist-test-slave-4h14"
server_key: "1a4b587b1d117703b8e033495633617f"
server_key_iv: "8c3d86b4ff2379d0f3a38d5d6aafef4d"
server_key_version: "encryptionkey@0"
I20260620 06:06:38.383471 26347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal/instance:
uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67"
format_stamp: "Formatted at 2026-06-20 06:06:38 on dist-test-slave-4h14"
server_key: "1a4b587b1d117703b8e033495633617f"
server_key_iv: "8c3d86b4ff2379d0f3a38d5d6aafef4d"
server_key_version: "encryptionkey@0"
I20260620 06:06:38.387458 26347 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260620 06:06:38.390138 26362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 06:06:38.391443 26347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260620 06:06:38.391623 26347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data,/tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal
uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67"
format_stamp: "Formatted at 2026-06-20 06:06:38 on dist-test-slave-4h14"
server_key: "1a4b587b1d117703b8e033495633617f"
server_key_iv: "8c3d86b4ff2379d0f3a38d5d6aafef4d"
server_key_version: "encryptionkey@0"
I20260620 06:06:38.391752 26347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 06:06:38.410163 26347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 06:06:38.414343 26347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 06:06:38.414613 26347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 06:06:38.422612 26347 rpc_server.cc:307] RPC server started. Bound to: 127.3.115.60:46783
I20260620 06:06:38.422922 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.115.60:46783 every 8 connection(s)
I20260620 06:06:38.423761 26347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/data/info.pb
I20260620 06:06:38.425243 3532 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu as pid 26347
I20260620 06:06:38.425421 3532 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskVS9eKZ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781935311537307-3532-0/minicluster-data/master-2/wal/instance
I20260620 06:06:38.425731 3532 external_mini_cluster.cc:1428] Setting key 30617251373b5d2992ca19637c194b55
I20260620 06:06:38.427503 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 06:06:38.432363 26264 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } attempt: 3
I20260620 06:06:38.433012 26415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.62@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
I20260620 06:06:38.441327 26339 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } attempt: 3
I20260620 06:06:38.442672 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.432907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34410)
0620 06:06:38.433069 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.433087 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.433187 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.433648 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.433652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.433855 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.433964 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.433971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.435048 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.435054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.435921 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.435938 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.436581 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.436584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.436719 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.437234 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.437255 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.438589 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.440747 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.440753 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.440759 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.440951 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.441303 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.441310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.441312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.441425 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.442143 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.442147 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.442247 (+ 100us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.442333 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.442490 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
I20260620 06:06:38.444528 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.427243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34406 (local address 127.3.115.60:46783)
0620 06:06:38.427750 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.427769 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.427827 (+ 58us) server_negotiation.cc:407] Connection header received
0620 06:06:38.428747 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.428771 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.429363 (+ 592us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.429785 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.430556 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.432100 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.433631 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.434139 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.434316 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.437942 (+ 3626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.437949 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.437968 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.438000 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.440335 (+ 2335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.440895 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.440901 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.440909 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.441034 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.441375 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.441379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.441380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.442017 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.442215 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.442991 (+ 776us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.443323 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":350,"spinlock_wait_cycles":1408,"thread_start_us":181,"threads_started":1}
I20260620 06:06:38.444528 26419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.433010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34410 (local address 127.3.115.60:46783)
0620 06:06:38.433376 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.433386 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.433411 (+ 25us) server_negotiation.cc:407] Connection header received
0620 06:06:38.433448 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.433451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.433494 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.434075 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.434186 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.434930 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.436058 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.436414 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.436575 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.438739 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.438744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.438747 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.438772 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.440606 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.441074 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.441080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.441082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.441170 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.441531 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.441534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.441536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.442017 (+ 481us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.443352 (+ 1335us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.443431 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.443554 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":294,"spinlock_wait_cycles":1920,"thread_start_us":151,"threads_started":1}
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.61@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
I20260620 06:06:38.448391 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.435214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.3.115.62:43365)
0620 06:06:38.435354 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.435358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.436295 (+ 937us) server_negotiation.cc:407] Connection header received
0620 06:06:38.436449 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.436454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.436514 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.436596 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.437668 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.438415 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.439838 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.440059 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.440175 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.443713 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.443717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.443719 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.443742 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.446262 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.446838 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.446842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.446844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.446926 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.447243 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.447249 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.447253 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.447566 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.447648 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.448103 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.448226 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 06:06:38.449309 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.435176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.62:43365 (local address 127.0.0.1:34242)
0620 06:06:38.436154 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.436206 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.436324 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.436777 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.436784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.437357 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.437528 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.437541 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.438560 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.438563 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.439508 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.439518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.440177 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.440181 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.440303 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.441154 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.441167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.443571 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.446363 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.446374 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.446384 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.446709 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.447047 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.447050 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.447052 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.447135 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.447740 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.447747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.447867 (+ 120us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.448838 (+ 971us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.449080 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":801,"thread_start_us":117,"threads_started":1}
I20260620 06:06:38.450027 26264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2: Bootstrap starting.
I20260620 06:06:38.450587 26415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:38.452138 26341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.441792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34420)
0620 06:06:38.442291 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.442309 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.442402 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.442939 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.442943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.443145 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.443258 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.443264 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.444977 (+ 1713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.444980 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.445729 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.445739 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.446216 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.446219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.446306 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.446767 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.446786 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.448315 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.450432 (+ 2117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.450435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.450438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.450703 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.451039 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.451043 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.451045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.451142 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.451601 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.451604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.451711 (+ 107us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.451822 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.451973 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393}
I20260620 06:06:38.452159 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.442001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34420 (local address 127.3.115.60:46783)
0620 06:06:38.442677 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.442685 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.442697 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:38.442733 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.442738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.442788 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.443368 (+ 580us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.443455 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.444835 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.445852 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.446101 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.446214 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.448466 (+ 2252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.448472 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.448475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.448498 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.450302 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.450828 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.450831 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.450833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.450927 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.451264 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.451267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.451268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.451485 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.451606 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.451883 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.452005 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":120,"threads_started":1}
I20260620 06:06:38.453820 26264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:38.454984 26264 log.cc:824] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:38.455310 26339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: Bootstrap starting.
I20260620 06:06:38.457350 26264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2: No bootstrap required, opened a new log
I20260620 06:06:38.458127 26339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:38.458969 26339 log.cc:824] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: Log is configured to *not* fsync() on all Append() calls
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.62@KRBTEST.COM
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:06:38.461058 26264 raft_consensus.cc:348] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.461354 26264 raft_consensus.cc:729] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b35ad3c107624560a57502ed402c34c2, State: Initialized, Role: FOLLOWER
I20260620 06:06:38.461750 26339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: No bootstrap required, opened a new log
I20260620 06:06:38.461748 26264 consensus_queue.cc:260] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.463302 26264 sys_catalog.cc:564] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:38.465040 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.451069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.61:44793 (local address 127.0.0.1:35310)
0620 06:06:38.451394 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.451420 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.451484 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.452755 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.452758 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.452926 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.453048 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.453056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.454094 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.454098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.455030 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.455041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.455725 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.455729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.455829 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.456306 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.456320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.460805 (+ 4485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.463056 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.463059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.463062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.463342 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.463685 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.463688 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.463690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.463783 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.464356 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.464359 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.464505 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.464612 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.464807 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":116,"threads_started":1}
I20260620 06:06:38.465735 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.451965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.3.115.61:44793)
0620 06:06:38.452083 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.452088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.452101 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:38.452133 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.452136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.452175 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.452253 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.453187 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.453955 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.455162 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.455507 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.455580 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.460980 (+ 5400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.460984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.460987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.461013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.462905 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.463484 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.463488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.463491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.463578 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.463936 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.463939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.463941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.464220 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.464311 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.465440 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.465574 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 06:06:38.466010 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.447777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34256 (local address 127.3.115.62:43365)
0620 06:06:38.448324 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.448336 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.448349 (+ 13us) server_negotiation.cc:407] Connection header received
0620 06:06:38.448383 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.448387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.448435 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.449843 (+ 1408us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.449957 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.450743 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.452402 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.453332 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.455275 (+ 1943us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.461598 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.461605 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.461611 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.461645 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.463547 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.464101 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.464104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.464107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.464207 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.464559 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.464563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.464566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.464832 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.465035 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.465696 (+ 661us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.465825 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":75,"threads_started":1}
I20260620 06:06:38.462720 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.466231 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.466820 26415 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } has no permanent_uuid. Determining permanent_uuid...
I20260620 06:06:38.467428 26339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.467706 26339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c3347eeabcc423f94272f28f68dae92, State: Initialized, Role: FOLLOWER
I20260620 06:06:38.468099 26339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.468627 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.468792 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.469683 26339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: configured and running, proceeding with master startup.
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935598, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.115.60@KRBTEST.COM for kudu/127.3.115.60@KRBTEST.COM
Jun 20 06:06:38 dist-test-slave-4h14 krb5kdc[26183](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781935597, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.115.61@KRBTEST.COM
I20260620 06:06:38.476110 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.467316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.115.60:46783 (local address 127.0.0.1:34426)
0620 06:06:38.467445 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0620 06:06:38.467460 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 06:06:38.467534 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 06:06:38.467911 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 06:06:38.467914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 06:06:38.468094 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 06:06:38.468169 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.468174 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.469144 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.469149 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.469714 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 06:06:38.469722 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.470199 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.470202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 06:06:38.470272 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.470790 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 06:06:38.470805 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 06:06:38.472629 (+ 1824us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 06:06:38.474582 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.474584 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.474587 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.474810 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.475077 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 06:06:38.475080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 06:06:38.475081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 06:06:38.475183 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 06:06:38.475617 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 06:06:38.475620 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 06:06:38.475694 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 06:06:38.475778 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 06:06:38.475928 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260620 06:06:38.476918 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.467627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34426 (local address 127.3.115.60:46783)
0620 06:06:38.467738 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.467742 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.467754 (+ 12us) server_negotiation.cc:407] Connection header received
0620 06:06:38.467786 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.467788 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.467817 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.467910 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.468275 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.469029 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.469856 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.470092 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.470199 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.472812 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.472819 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.472823 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.472851 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.474464 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.474905 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.474907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.474909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.474978 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.475279 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.475282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.475284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.475501 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.475587 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.476634 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.476784 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 06:06:38.477126 26441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260620 06:06:38.480180 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: Bootstrap starting.
I20260620 06:06:38.480710 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.467247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.3.115.61:44793)
0620 06:06:38.467352 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.467356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.467529 (+ 173us) server_negotiation.cc:407] Connection header received
0620 06:06:38.467566 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.467569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.467613 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.467677 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.468558 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.469246 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.470273 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.470611 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.470727 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.476556 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.476561 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.476563 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.476589 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.478318 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.478870 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.478874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.478876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.478961 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.479292 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.479295 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.479297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.479514 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.479704 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.480406 (+ 702us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.480518 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 06:06:38.479195 26271 master_runner.cc:186] Error getting master registration for 127.3.115.60:46783: OK, instance_id { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" instance_seqno: 1781935598420190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:06:38.482561 26271 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 06:06:38.483263 26415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: Neither blocks nor log segments found. Creating new log.
I20260620 06:06:38.484016 26415 log.cc:824] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: Log is configured to *not* fsync() on all Append() calls
I20260620 06:06:38.484856 3532 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 06:06:38.486016 26238 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "b35ad3c107624560a57502ed402c34c2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34256
I20260620 06:06:38.486281 26238 raft_consensus.cc:482] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 06:06:38.486402 26238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 0 FOLLOWER]: Advancing to term 1
W20260620 06:06:38.486784 26449 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:38.486860 26449 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:38.487635 26238 raft_consensus.cc:504] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.477798 26196 master_runner.cc:186] Error getting master registration for 127.3.115.60:46783: OK, instance_id { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" instance_seqno: 1781935598420190 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 06:06:38.488205 26196 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 06:06:38.477196 26441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:38.488628 26238 leader_election.cc:288] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [CANDIDATE]: Term 1 election: Requested vote from peers 4c3347eeabcc423f94272f28f68dae92 (127.3.115.61:44793), 8fb7e9b0510a4a049dc98fb2e0e4cb67 (127.3.115.60:46783)
I20260620 06:06:38.489089 26313 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b35ad3c107624560a57502ed402c34c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4c3347eeabcc423f94272f28f68dae92"
I20260620 06:06:38.489336 26313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:38.490285 26415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: No bootstrap required, opened a new log
I20260620 06:06:38.490799 26313 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35ad3c107624560a57502ed402c34c2 in term 1.
I20260620 06:06:38.491323 26214 leader_election.cc:302] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c3347eeabcc423f94272f28f68dae92, b35ad3c107624560a57502ed402c34c2; no voters:
I20260620 06:06:38.491708 26426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 06:06:38.491977 26426 raft_consensus.cc:686] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [term 1 LEADER]: Becoming Leader. State: Replica: b35ad3c107624560a57502ed402c34c2, State: Running, Role: LEADER
I20260620 06:06:38.492345 26426 consensus_queue.cc:237] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.494445 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b35ad3c107624560a57502ed402c34c2. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.494594 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:38.495398 26459 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 06:06:38.495330 26415 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.495666 26415 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fb7e9b0510a4a049dc98fb2e0e4cb67, State: Initialized, Role: FOLLOWER
I20260620 06:06:38.496037 26415 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } }
I20260620 06:06:38.497598 26415 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 06:06:38.497668 26389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b35ad3c107624560a57502ed402c34c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67"
I20260620 06:06:38.497921 26389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [term 0 FOLLOWER]: Advancing to term 1
I20260620 06:06:38.498379 26459 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 06:06:38.499387 26389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b35ad3c107624560a57502ed402c34c2 in term 1.
W20260620 06:06:38.505046 26471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 06:06:38.505203 26471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 06:06:38.513386 26422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.496749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.3.115.62:43365)
0620 06:06:38.496869 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.496874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.503387 (+ 6513us) server_negotiation.cc:407] Connection header received
0620 06:06:38.503599 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.503603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.503659 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.503709 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.504886 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.505652 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.506900 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.507309 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.507392 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.509460 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.509464 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.509467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.509500 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.511301 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.511854 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.511857 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.511859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.511949 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.512285 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.512289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.512290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.512529 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.512650 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.513052 (+ 402us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.513198 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 06:06:38.516857 26460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.517037 26460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.521530 26313 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [term 1 FOLLOWER]: Refusing update from remote peer b35ad3c107624560a57502ed402c34c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:38.523065 26389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [term 1 FOLLOWER]: Refusing update from remote peer b35ad3c107624560a57502ed402c34c2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 06:06:38.523554 26426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:38.528754 26426 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 06:06:38.533701 26460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b35ad3c107624560a57502ed402c34c2. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.533886 26460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.534505 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b35ad3c107624560a57502ed402c34c2. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.534662 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.539544 26456 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.539812 26456 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:38.540707 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.503455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35328 (local address 127.3.115.61:44793)
0620 06:06:38.503594 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.503598 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.513963 (+ 10365us) server_negotiation.cc:407] Connection header received
0620 06:06:38.514197 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.514201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.514258 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.514313 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.515150 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.515914 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.517194 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.517482 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.517560 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.535332 (+ 17772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.535337 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.535340 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.535374 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.537473 (+ 2099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.539248 (+ 1775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.539252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.539255 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.539353 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.539668 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.539672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.539674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.539903 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.540031 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.540391 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.540525 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 06:06:38.541545 26436 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.541699 26436 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.543408 26477 mvcc.cc:205] Tried to move back new op lower bound from 7298808211534508032 to 7298808211428917248. Current Snapshot: MvccSnapshot[applied={T|T < 7298808211534508032}]
I20260620 06:06:38.545311 26421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:38.502736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.3.115.60:46783)
0620 06:06:38.502845 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.502851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.512965 (+ 10114us) server_negotiation.cc:407] Connection header received
0620 06:06:38.513165 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.513170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.513227 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.513279 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.515662 (+ 2383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.516433 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.534026 (+ 17593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:38.534409 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:38.534496 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:38.541216 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 06:06:38.541223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 06:06:38.541229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 06:06:38.541265 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0620 06:06:38.543085 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.543630 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.543634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.543636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.543734 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 06:06:38.544055 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 06:06:38.544059 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 06:06:38.544061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0620 06:06:38.544287 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 06:06:38.544444 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:38.544878 (+ 434us) server_negotiation.cc:299] Negotiation successful
0620 06:06:38.545078 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 06:06:38.546442 26459 catalog_manager.cc:1347] Generated new cluster ID: 8886fe15392b4775b5da8bcbeea723e0
I20260620 06:06:38.546517 26459 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 06:06:38.547578 26426 sys_catalog.cc:455] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.547712 26426 sys_catalog.cc:458] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2 [sys.catalog]: This master's current role is: LEADER
I20260620 06:06:38.549823 26479 mvcc.cc:205] Tried to move back new op lower bound from 7298808211534508032 to 7298808211428917248. Current Snapshot: MvccSnapshot[applied={T|T < 7298808211534508032}]
I20260620 06:06:38.552814 26460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b35ad3c107624560a57502ed402c34c2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b35ad3c107624560a57502ed402c34c2" member_type: VOTER last_known_addr { host: "127.3.115.62" port: 43365 } } peers { permanent_uuid: "4c3347eeabcc423f94272f28f68dae92" member_type: VOTER last_known_addr { host: "127.3.115.61" port: 44793 } } peers { permanent_uuid: "8fb7e9b0510a4a049dc98fb2e0e4cb67" member_type: VOTER last_known_addr { host: "127.3.115.60" port: 46783 } } }
I20260620 06:06:38.552953 26460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 06:06:38.567677 26459 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 06:06:38.569090 26459 catalog_manager.cc:1504] Loading token signing keys...
I20260620 06:06:38.586131 26459 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b35ad3c107624560a57502ed402c34c2: Generated new TSK 0
I20260620 06:06:38.587180 26459 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 06:06:38.621932 26422 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.618240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34266 (local address 127.3.115.62:43365)
0620 06:06:38.618375 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.618381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.620470 (+ 2089us) server_negotiation.cc:407] Connection header received
0620 06:06:38.620948 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.620952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.621011 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.621070 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.621784 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34266: BlockingRecv error: recv got EOF from 127.0.0.1:34266 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260620 06:06:38.623379 26421 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.619082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34452 (local address 127.3.115.60:46783)
0620 06:06:38.619201 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.619206 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.620856 (+ 1650us) server_negotiation.cc:407] Connection header received
0620 06:06:38.622558 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.622561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.622615 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.622667 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.623234 (+ 567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34452: BlockingRecv error: recv got EOF from 127.0.0.1:34452 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 06:06:38.624562 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.619819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.3.115.61:44793)
0620 06:06:38.619932 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.619936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.622350 (+ 2414us) server_negotiation.cc:407] Connection header received
0620 06:06:38.623762 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.623766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.623818 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.623870 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.624422 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35338: BlockingRecv error: recv got EOF from 127.0.0.1:35338 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260620 06:06:38.629710 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.628033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35352 (local address 127.3.115.61:44793)
0620 06:06:38.628143 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.628147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.628595 (+ 448us) server_negotiation.cc:407] Connection header received
0620 06:06:38.628906 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.628910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.628991 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.629046 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.629589 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35352: BlockingRecv error: recv got EOF from 127.0.0.1:35352 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 06:06:38.635303 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.633587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.3.115.61:44793)
0620 06:06:38.633710 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.633715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.634289 (+ 574us) server_negotiation.cc:407] Connection header received
0620 06:06:38.634484 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.634488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.634557 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.634614 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.635162 (+ 548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35364: BlockingRecv error: recv got EOF from 127.0.0.1:35364 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260620 06:06:38.641144 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.639468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35374 (local address 127.3.115.61:44793)
0620 06:06:38.639581 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.639585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.640141 (+ 556us) server_negotiation.cc:407] Connection header received
0620 06:06:38.640331 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.640336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.640390 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.640444 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.641025 (+ 581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35374: BlockingRecv error: recv got EOF from 127.0.0.1:35374 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260620 06:06:38.649374 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.647428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.3.115.61:44793)
0620 06:06:38.647593 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.647597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.648293 (+ 696us) server_negotiation.cc:407] Connection header received
0620 06:06:38.648503 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.648506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.648578 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.648637 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.649249 (+ 612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35390: BlockingRecv error: recv got EOF from 127.0.0.1:35390 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260620 06:06:38.661877 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.659968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35400 (local address 127.3.115.61:44793)
0620 06:06:38.660098 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.660102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.660816 (+ 714us) server_negotiation.cc:407] Connection header received
0620 06:06:38.661040 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.661044 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.661105 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.661161 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.661748 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35400: BlockingRecv error: recv got EOF from 127.0.0.1:35400 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260620 06:06:38.682312 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.680579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35402 (local address 127.3.115.61:44793)
0620 06:06:38.680726 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.680730 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.681324 (+ 594us) server_negotiation.cc:407] Connection header received
0620 06:06:38.681541 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.681545 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.681599 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.681650 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.682191 (+ 541us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35402: BlockingRecv error: recv got EOF from 127.0.0.1:35402 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260620 06:06:38.718415 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.717256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.3.115.61:44793)
0620 06:06:38.717376 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.717380 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.717492 (+ 112us) server_negotiation.cc:407] Connection header received
0620 06:06:38.717671 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.717674 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.717726 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.717821 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.718278 (+ 457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35416: BlockingRecv error: recv got EOF from 127.0.0.1:35416 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260620 06:06:38.786185 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.784846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.3.115.61:44793)
0620 06:06:38.785029 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.785036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.785225 (+ 189us) server_negotiation.cc:407] Connection header received
0620 06:06:38.785414 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.785418 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.785476 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.785588 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.786070 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35424: BlockingRecv error: recv got EOF from 127.0.0.1:35424 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260620 06:06:38.918206 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:38.916881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35426 (local address 127.3.115.61:44793)
0620 06:06:38.917085 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:38.917090 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:38.917270 (+ 180us) server_negotiation.cc:407] Connection header received
0620 06:06:38.917451 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:38.917454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:38.917507 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:38.917581 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:38.918033 (+ 452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35426: BlockingRecv error: recv got EOF from 127.0.0.1:35426 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 06:06:39.177762 26345 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 06:06:39.176728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.3.115.61:44793)
0620 06:06:39.176905 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:39.176910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:39.176929 (+ 19us) server_negotiation.cc:407] Connection header received
0620 06:06:39.176983 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:39.176985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:39.177030 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:39.177123 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0620 06:06:39.177615 (+ 492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35428: BlockingRecv error: recv got EOF from 127.0.0.1:35428 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 06:06:39.488253 26449 catalog_manager.cc:1259] Loaded cluster ID: 8886fe15392b4775b5da8bcbeea723e0
I20260620 06:06:39.488358 26449 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: loading cluster ID for follower catalog manager: success
I20260620 06:06:39.490156 26449 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: acquiring CA information for follower catalog manager: success
I20260620 06:06:39.491441 26449 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4c3347eeabcc423f94272f28f68dae92: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:06:39.506565 26471 catalog_manager.cc:1259] Loaded cluster ID: 8886fe15392b4775b5da8bcbeea723e0
I20260620 06:06:39.506659 26471 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: loading cluster ID for follower catalog manager: success
I20260620 06:06:39.508447 26471 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: acquiring CA information for follower catalog manager: success
I20260620 06:06:39.509699 26471 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8fb7e9b0510a4a049dc98fb2e0e4cb67: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 06:06:39.696142 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:39.691947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35444 (local address 127.3.115.61:44793)
0620 06:06:39.692210 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:39.692213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:39.692318 (+ 105us) server_negotiation.cc:407] Connection header received
0620 06:06:39.692452 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:39.692454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:39.692600 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:39.692747 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:39.693049 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.693577 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.694457 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.694702 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.694790 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:39.695210 (+ 420us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:39.695672 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:39.695774 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:39.695869 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 06:06:39.695926 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":101,"threads_started":1}
I20260620 06:06:39.703272 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:39.699594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34462 (local address 127.3.115.60:46783)
0620 06:06:39.699824 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:39.699827 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:39.699842 (+ 15us) server_negotiation.cc:407] Connection header received
0620 06:06:39.699987 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:39.699990 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:39.700112 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:39.700221 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:39.700449 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.701046 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.701864 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.702087 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.702174 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:39.702471 (+ 297us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:39.702938 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:39.703049 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:39.703101 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 06:06:39.703150 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260620 06:06:39.710811 26554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:39.706881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.3.115.60:46783)
0620 06:06:39.706998 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:39.707002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:39.707435 (+ 433us) server_negotiation.cc:407] Connection header received
0620 06:06:39.707574 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:39.707577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:39.707625 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:39.707699 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:39.707999 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.708744 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.709600 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.709919 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.710027 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:39.710312 (+ 285us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:39.710410 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:39.710493 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:39.710578 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 06:06:39.710647 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 06:06:39.711220 26548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:39.706598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35460 (local address 127.3.115.61:44793)
0620 06:06:39.706749 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:39.706756 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:39.706986 (+ 230us) server_negotiation.cc:407] Connection header received
0620 06:06:39.707197 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:39.707201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:39.707263 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:39.707338 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:39.707657 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.708419 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.709783 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.710082 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.710181 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:39.710467 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:39.710560 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:39.710625 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:39.711037 (+ 412us) server_negotiation.cc:299] Negotiation successful
0620 06:06:39.711100 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 06:06:39.712649 26561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 06:06:39.706507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.3.115.62:43365)
0620 06:06:39.706787 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 06:06:39.706792 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 06:06:39.707201 (+ 409us) server_negotiation.cc:407] Connection header received
0620 06:06:39.707304 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 06:06:39.707310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 06:06:39.707481 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 06:06:39.707618 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 06:06:39.708755 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.709590 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.710684 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 06:06:39.711046 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 06:06:39.711137 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 06:06:39.711831 (+ 694us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 06:06:39.712217 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 06:06:39.712296 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 06:06:39.712389 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 06:06:39.712450 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":84,"threads_started":1}
I20260620 06:06:39.716468 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 26196
I20260620 06:06:39.723176 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 26271
I20260620 06:06:39.729449 3532 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskVS9eKZ/build/debug/bin/kudu with pid 26347
2026-06-20T06:06:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3132 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6282 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (288191 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST